Skip to main content

Full text of "USPTO Patents Application 09775169"

See other formats


This Page Is Inserted by IFW Operations 
and is not a part of the Official Record 



BEST AVAILABLE IMAGES 



Defective images within this document are accurate representations of 
the original documents submitted by the applicant. 

Defects in the images may include (but are not hmited to): 



BLACK BORDERS 

TEXT CUT OFF AT TOP, BOTTOM OR SIDES 
FADED TEXT 
ILLEGIBLE TEXT 
SKEWED/SLANTED IMAGES 
COt.ORED PHOTOS 

BLACK OR VERY BLACK ANT) WHITE DARK PHOTOS 
CRAY SCALE DOCLiMENTS 



IMAGES ARE BEST AVAILABLE COPY. 

As rescanning documents n-/// not correct images, 
please do not report the images to the 
Image Problem Mailbox. 



illlllilliilliillllDiiilllliii 



US 20020003528A1 



(19) United States 

(12) Patent Application Publication (lo) Pub. No.: US 2002/0003528 Ai 

Rosenberg et al. (43) Pub. Date: Jan, 10, 2002 



(54) CURSOR CONTROL USING A TACTILE 
FEEDBACK DEVICE 

(75) Inveoiors: Louis B. Rosenberg, Pleasanton, CA 
(US); Jonathan L. Beamer, Menlo 
Park, CA (US); Adam C. Braun, 
Sunnyvale, CA (US); Dean C. Chang, 
Palo Alto, CA(US) 

Correspondence Address: 
James R. Riegel 

IMMERSION CORPORATION 

801 Fox Lane 

San Jose, CA 95131 (US) 

(73) Assignee: Immersion Corporation 

(21) Appl. No.: 09/935,102 

(22) Filed: Aug. 21, 2001 

Related U.S. Application Data 

(60) Continuation of application No. 09/343,940, filed on 
Jun. 30, 1999, now Pat. No. 6,288,705, which is a 
division of application No. 08/924,462, filed on Aug. 
23, 1997, now Pat. No. 6,252,579. 



Publication Classification 

(51) Int. CI.' G09G 5/08 

(52) U.S. CI 345/157 



(57) 



ABSTRACT 



A mouse interface device and method for providing 
enhanced cursor control and indexing cursor control with 
force feedback. A force feedback interface device includes a 
manipulandum, such as a mouse, that is moveable in a local 
workspace. The device is coupled to a host computer that 
displays a cursor in a graphical environment, such as a GUI, 
on a display screen. A cursor position in the display frame 
is reported to the host computer derived from a reference 
position of the mouse in the local frame, and the host 
displays the cursor; for example, the cursor position may be 
scaled by a ballistics algorithm based on mouse velocity to 
allow fine positioning or coarse motion of the cursor. A force 
is output on the mouse based on interactions in the GUI, the 
force being determined based on mouse reference data or 
cursor ballistic data, depending on the type of force, to 
reduce distortion between visual and force outputs. Assistive 
forces can alternatively be output to achieve the enhanced 
cursor control. Indexing features allow control of the cursor 
when an ofi&et between local and display frames exists, 
allow the user to reduce the offset, and reduce disconcerting 
collisions of the mouse with physical workspace limits. 




12/08/2003, EAST version: 1.4.1 



Patent Application Publication Jan. 10, 2002 Sheet 1 of 14 US 2002/0003528 Al 




12/08/2003, EAST version: 1.4.1 



Patent Application PubUcation Jan. 10, 2002 Sheet 2 of 14 US 2002/0003528 Al 




12/08/2003, EAST version: 1.4.1 




12/08/2003, EAST Version: 1.4.1 



J 

Patent AppUcation PubUcation Jan. 10, 2002 Sheet 4 of 14 US 2002/0003528 Al 



110 



RAM 1 


ROM 

1 


HOST 
PROCESSOR 



HOST COMPUTER 
112 



-—18 




AUDIO OUT- 
PUT DEVICE 




12/08/2003, EAST version: 1.4.1 



J 



Patent AppUcation PubUcation Jan. 10, 2002 Sheet 5 of 14 US 2002/0003528 Al 



LOCAL 



Q^jQ'N LOCAL FRAME (MO USE WORKSPACE) 
182 



30 




442 



SCREEN 
ORIGIN 

184 



Y_LOCAL 



440-^ 
DISPLAY FRAME 



! Y_SCREEN 



U -Cl 443 
X.SCREEN 

JiiLfso 

X_LOCAL 



440 




440 



440 



'Cs 




(SCREEN) 



442 




■442 



W 

■444 



I I 

I I 
I J 



401 



28 

J 



^403 



12/08/2003, EAST Version: 1.4.1 



A 

Patent Application PubUcation Jan. 10, 2002 Sheet 6 of 14 US 2002/0003528 Al 




202 



START 



200 



204 



READ MOUSE POSITION IN LOCAL 
FRAME AS REFERENCE POSITION 



DETERMINE BALLISTIC POSITION 
FROM REFERENCE DATA 



DETERMINE INTERACTIONS BETWEEN - 
CURSOR AND GUI USING BALLISTIC POSITION 



± 



CALCULATE INDEPENDENT FORCES 
— — — 



CALCULATE REMAINING FORCES 
USING REFERENCE DATA 



SUM AND OUTPUT FORCES 



218 



i 




206 



208 



210 



212 



214 



CURSOR 
POSITION = 
BALLISTIC 
POSITION 



220 



CURSOR POSITION 
BASED ON CONSTANT 
SCALE FACTOR 



REPORT CURSOR 
POSITION TO HOST 
COMPUTER 



222 



12/08/2003, EAST version: 1.4.1 



Patent AppUcation PubUcation Jan. 10, 2002 Sheet 7 of 14 US 2002/0003528 Al 
240-^242 



% 7a ^ 



180 



246 
12 



^if. 7 c ^ 



242 F/tSr 750 



244 




% 7d cif!!-^:^ 



249^ J 

12 

'242 ^ SLO W c — 250 

180 



. ^242 i^LOW ^ ii 50 



^-12 



12/08/2003, EAST version: 1.4.1 



Patent Application PubUcation Jan. 10, 2002 Sheet 8 of 14 US 2002/0003528 Al 



START 




302 




READ MOUSE POSITION IN LOCAL 
FRAME AS REFERENCE POSITION 



DETERMINE BALLISTIC POSITION 
FROM REFERENCE DATA 



DETERMINE INTERACTIONS BETWEEN 
CURSOR AND GUI USING BALLISTIC POSITION 



CALCULATE INDEPENDENT FORCES 



CALCULATE MOUSE-BASED FORCES 
USING REFERENCE DATA 

^ 



CALCULATE CURSOR-BASED FORCES 
USING REFERENCE DATA 

I 



306 



308 



310 



312 



314 



REPORT BALLISTIC POSITION TO HOST 
COMPUTER AS CURSOR POSITION 



316 



SUM AND OUTPUT FORCES 



318 



12/08/2003, EAST version: 1.4.1 



Patent Application PubUcation Jan. 10, 2002 Sheet 9 of 14 US 2002/0003528 Al 




408 



READ MOUSE POSITION IN LOCAL 
FRAME AS REFERENCE POSITION 



CALCUUTE 
BALLISTIC 
SCREEN 
FACTOR (BSF) 
BASED ON 

MOUSE 
VELOCITY 



414 



406 

, MOUSE 
NO WITHIN A PRE- "\ YES 

DETERMINED DISTANCE OF 
PHYSICAL WORKSPACE 
LIMIT? 

412 



CALCULATE 
ISOMETRIC RATE 
BASED ON 
PENETRATION 
OF ISOMETRIC 
REGION 



CHANGE IN 
CURSOR 
POSITION=BSF* 
CHANGE IN 
MOUSE POSITION 



47^ 



CURSOR POSITIONS 
CURSOR POSITION* 
ACURSOR POSITION 



CHANGE IN 
CURSOR POSITION 
=ISOMETRIC RATE 



DETERMINE INTERACTIONS BETWEEN 
CURSOR AND GUI USI NG CURSOR POSITION 

i 



t 



416 



426 



CALCULATE INDEPENDENT FORCES 

CALCUUTE CURSOR-BASED FORCES 
USING BALLISTIC DATA 



418 



420 




CALCULATE 
MOUSE-BASED 
FORCES USING 

REFERENCE 
DATA 



REPORT CURSOR 
POSITION TO HOST 
COMPUTER 



T 



CALCULATE 
EQUIVALENT 
FORCES TO 
SUBSTITUTE FOR 
MOUSE-BASED 
FORCES 



SUM AND OUTPUT 
FORCES 



■434 



430 



12/08/2003, EAST version: 1.4.1 



Patent Application PubUcation Jan. 10, 2002 Sheet 10 of 14 US 2002/0003528 Al 




12/08/2003, EAST version: 1.4.1 



Patent AppUcation PubUcation Jan. 10, 2002 Sheet 11 of 14 US 2002/0003528 Al 



502 




START 



5Cf4 



r 



500 



READ MOUSE POSITION IN LOCAL 
FRAME AS REFERENCE POSITION 



NO 



508 



1 



506 

MOUSE^ 
WITHIN A PRE- 
DETERMINED DISTANCE 
OF PHYSICAL WORKSPACE 
UMITAND MOVING 
TOWARD LIMIT? 



YES 



514 

L 



DETERMINE DISTANCE 
BETWEEN CURRENT 
POSITION OF MOUSE 
AND CLOSEST 
WORKSPACE LIMIT 



CALCULATE 
BALLISTIC 
SCREEN 
FACTOR (BSF) 
BASED ON MOUSE 
VELOCITY 



516- 



518- 



DETERMINE DISTANCE BETWEEN 
CURRENT POSITION OF 
CURSOR AND SCREEN LIMIT 



T 



CHANGE IN 
CURSOR 
POSITION=BSP 
CHANGE IN 
MOUSE POSITION 

510 



512 



DETERMINE SCALING FACTOR 
FOR CURSOR POSITION USING 
DISTANCES 



CURSOR POSmON= 
CURSOR P0SITI0N4 
ACURSOR POSITION 



CHANGE IN 
CURSOR POSITION 

BASED ON 
SCALING FACTOR 



DETERMINE INTERACTIONS BETWEEN 
CURSER AND GUI USING CURSOR POSITION 



520 



CALCULATE INDEPENDENT FORCES 



CALCULATE MOUSE-BASED FORCES 
USING REFERENCE DATA 



CALCULATE CURSOR-BASED FORCES 
USING REFERENCE DATA 



REPORT CURSOR POSITION TO HOST COMPUTER 



SUM AND OUTPUT FORCES 



522 
524 

-526 

-528 

-530 
532 



12/08/2003, EAST Version: 1.4.1 



Patent Application PubUcation Jan. 10, 2002 Slieet 12 of 14 US 2002/0003528 Al 

600 

602 




. . . J; """"^"^ nt=i>ib I IVE FORCE 
MAGNITUDE INVERSELY BASED 
ON MOUSE VELOCITY 



SEND REFERENCE POSITION 
TO HOST COMPUTER AS 
CURSOR POSITION 





READ MOUSE POSITION IN MOUSE 
FRAMEAS REFERENCE POSITION ^ 


-^604 






r 






EXAMINE PREVIOUS POSITIONS 
OF MOUSE TO DETERMINE 
VELOCITY OF MOUSE 


■^606 




> 


r 





608 




610 



612 



12/08/2003, EAST version: 1.4.1 



Patent AppUcation PubUcation Jan. 10, 2002 Sheet 13 of 14 US 2002/0003528 Al 




START 



622 



620 



624 



L.^f$f '^^ ^^^^ LEVELCOMMAND FROM 
^IS'^^"^"^^ LOCATION OF DETENTfS) 
IN LOCAL FRAME AND PARAMETERS 
DESCRIBING DETENT 




OUTPUT DETENT 



FORCE 



SEND REFERENCE 
POSITION TO HOST 

COMPUTER AS 
CURSOR POSITION 



632 



12/08/2003, EAST version: 1.4.1 



Patent AppUcation PubUcation Jan. 10, 2002 Sheet 14 of 14 US 2002/0003528 Al 



,650 



START 




652 



READ MOUSE POSITION IN MOUSE 
FRAME AS REFERENCE POSITION 



EXAMINE PREVIOUS 
POSITIONS OF MOUSE 
TO DETERMINE ■ 
VELOCITY OF MOUSE 



654 



656 



658 



MOUSE UNDER 
THRESHOLD VELOCITY 
FOR > PREDETERMINED 
TIME PERIOD? 



YES 



PROVIDE DETENTS IN A 
DETERMINED SPACING 
OVER A DETERMINED AREA 




660 



662 



SEND REFERENCE 
POSITION TO HOST 

COMPUTER AS 
CURSOR POSITION 



666 



12/08/2003, EAST Version: 1.4.1 



us 2002/0003528 Al 



1 



Jan. 10, 2002 



CURSOR CONTROL USING A TACTILE 
FEEDBACK DEVICE 

BACKGROUND OF mE INVENTION 

[0001] The present invention relates generally to interface 
devices for allowing humans to interface with computer 
systems, and more particularly to computer interface devices 
that allow the user to provide input to computer systems and 
provide force feedback to the user. 

[0002] Computer systems are used extensively to imple- 
ment many applications, such as word processing, data 
management, simulations, games, and other tasks. A com- 
puter system typically displays a visual environment to a 
user on a display screen or other visual output device. Users 
can interact with the displayed environment to perform 
functions on the computer, play a game, experience a 
simulated environment, use a computer aided design (CAD) 
system, etc. One visual environment that is particularly 
common is a graphical user interface (GUI). GUI's present 
visual images which describe various graphical metaphors 
of a program or operating system implemented on the 
computer. Common GUI's include the Windows™ operat- 
ing system from Microsoft Corporation and the MacOS 
operating system from Apple Computer, Inc. The user 
typically moves a displayed, user-controlled graphical 
object, such as a cursor or pointer, across a computer screen 
and onto other displayed graphical objects or predefined 
screen regions, and then inputs a command to execute a 
given selection or operation. The objects or regions ("tar- 
gets") can include, for example, icons, windows, pull -down 
menus, buttons, and scroll bars. Most GUI's are currently 
2-dimensional as displayed on a computer screen; however, 
three dimensional (3-D) GUI's that present simulated 3-D 
environments on a 2-D screen can also be provided. Other 
programs or environments that may provide user-controlled 
graphical objecLs such as a cursor or a "view" controlled by 
the u.ser include graphical "web pages" or other environ- 
ments offered on the World Wide Web of the Internet, CAD 
programs, video games, virtual reality simulations, etc. 

[0003] The user interaction with and manipulation of the 
computer environment is achieved using any of a variety of 
types of human-computer interface devices that are con- 
nected to the computer system controlling the displayed 
environment. In most systems, the computer updates the 
environment in response to the user's manipulation of a 
user-manipulatable physical object ("user object") that is 
included in the interface device, such as a mouse, joystick, 
etc. 'l^e computer provides feedback to the user utilizing the 
display screen and, typically, audio speakers. 

[0004] A computer mouse is a common user object used to 
interact with a GUI or other graphical environment. A mouse 
(and other mouse-type devices such as a track ball) is 
typically used as a position control device in which dis- 
placement of the mouse in a planar workspace (e.g. on a 
mouse pad) is directly correlated to displacement of the 
user-controlled graphical object, such as a cursor, displayed 
on the screen. This displacement correlation may not be a 
one-to-one correspondence, since the cursor position may be 
scaled according to a constant mapping from the mouse 
position e.g., the mouse may be moved a distance of one 
inch on a mouse pad which causes the controlled cursor to 
move four inches across the screen. In most cases, small 



movements of the mouse are scaled to large motions of the 
cursor on the screen to allow the user to easily point to 
targets in all areas of the screen. The user can typically 
change the scaling or "pointer speed'* of a cursor to a desired 
level, which is the ratio or scaling factor of cursor movement 
to mouse movement, using menus provided in the operating 
system or application program. 

[0005] The scaled cursor movement in a GUI works well 
for coarse cursor motion, which is the broad, sweeping 
motion of the cursor that brings the cursor from one global 
area on the screen to another. Accuracy of cursor motion is 
not critical for coarse motion, but speed of the cursor is 
ideally, the cursor traverses the desired distance on the 
screen quickly and efficiently. For such tasks, it is valuable 
for the cursor to move a large distance with small motions 
of the physical mouse hardware. However, a problem occurs 
in mouse -type devices when the user wishes to move the 
cursor a short distance or in small increments ("fine posi- 
tioning"). For tasks in which accurate positioning of the 
cursor is needed, such as target acquisition tasks, the large 
scaling of mouse movement to cursor movement is inad- 
equate or even harmful. For example, the user may wish to 
move the cursor onto a GUI target such as an icon or menu 
item. If very small motions of the mouse result in large 
cursor motion, the user may simply lack the manual dex- 
terity to acquire the target. Certain target acquisition tasks 
where the targets are very small can be particularly chal- 
lenging even if the mapping between the cursor and the 
mouse is reasonable for most other cursor motion activities. 
For example, in drawing programs it is often required that a 
user position the cursor on a very small "point" or "node" on 
the screen; and in some cases, the target can be as small as 
a single display pixel. For such situations, a scaling that 
causes large motions of the cursor for small motions of the 
mouse may make a target acquisition task physically impos- 
sible for the user. 

[0006] Mouse "ballistics" or "ballistic tracking" is typi- 
cally used to alleviate the scaling problem for fine position- 
ing of the cursor. Ballistics refers to the technique of varying 
the scaling between motion of a physical mouse and motion 
of a displayed cursor depending upon the velocity of the 
mouse in its workspace. The assumption is that if the user is 
moving the mouse very quickly, the user is likely performing 
a "coarse motion" task on the screen, and therefore the 
mouse driver scales smaU motions of the mouse to large 
motions of the cursor. Conversely, if the user is moving the 
mouse very slowly, the user is likely performing a fine 
positioning task on the screen, and the mouse driver scales 
small motions of the mouse to small motions of the cursor. 
Such a variable scaling technique is disclosed in U.S. Pat. 
No. 4,734,685 of Watanabe and 5,195,179 of Tokunaga. 

[0007] Many algorithms can be used for mouse ballistics. 
The simplest method is to designate a threshold velocity 
such that if the mouse is moving faster than the threshold 
velocity, a large scaling of cursor position is made so that 
small motions of the mouse cause large motions of the 
cursor; and if the mouse is moving slower than the threshold 
velocity, a smaller scaling is made so that small motions of 
the mouse cause small motions of the cursor. A more 
sophisticated and more common method is to gradually 
change the scaling in accordance with mouse velocity using 
a continuous function. This can be a simple linear function, 
such as a direction relation between mouse speed and the 



12/08/2003. EAST version: 1.4.1 



us 2002/0003528 Al 



2 



Jan. 10, 2002 



distance the cursor moves for a given increment of mouse 
motion, or a nonlinear function that is optimized in a 
particular way. The "mapping** of the cursor to the mouse is 
the method of translating the mouse position in its work- 
space to a cursor position on the display screen and may 
involve ballistics or other algorithms and scale factors. 

[0008] A problem occurs when standard ballistics tech- 
niques are used with force feedback interface devices. Force 
feedback interface devices allow a user to experience forces 
on the manipulated user object based on interactions and 
events within the displayed graphical environment. Typi- 
cally, computer-controlled motors or other actuators are used 
to output forces on the user object in provided degrees of 
freedom to simulate various sensations, such as an obstruc- 
tion force when moving the cursor into a wall, a damping 
force to"^ resist motion of the cursor, and a spring force to bias 
the cursor to move back toward a starting position of the 
spring. 

[0009] Force feedback devices can be implemented in 
many forms, such as a joystick, mouse, steering wheel, etc. 

[0010] When these and other types of forces are imple- 
mented in conjunction with mouse ballistics, a conflict 
occurs between the use of ballistics position and force 
feedback output. In general, force feedback is generated 
based directly on motion of the mouse while visual feedback 
such as movement of the cursor does not correspond directly 
with motion of the mouse due to scaling and ballistics. As 
explained above, 1 mm displacement of the mouse may 
cause different visual results in ciu-sor motion based on the 
mouse velocity when using ballistics. Thus, when imple- 
menting forces in an interface device, motion of the mouse 
can no longer be consistently correlated to cursor motion on 
the screen due to the variable scaling. This is a particular 
problem for a force feedback mouse system because in 
general, feel sensations such as springs, surfaces, dampers, 
textures, masses, and other spatially related physical phe- 
nomenon rely on a constant, predictable mapping between 
the motion of the mechanical object (mouse), the forces 
generated on the mouse, and graphically displayed interac- 
tions on the screen (cursor motion). 

[0011] For example, a spring sensation can be used when 
the user manipulates a cursor to stretch a graphically dis- 
played element on the screen, such as a line in a drawing 
program or a window in a GUI. The user positions the cursor 
on the line, and moves the cursor to stretch the line. Visually, 
this stretch is displayed based on cursor motion. In addition, 
the accompanying spring sensation outputs a resistance 
force that increases linearly with displacement. Traditional 
force feedback systems would use displacement of the 
mouse in its workspace as the displacement magnitude 
required to calculate the spring force. If no ballistics are in 
effect, no problem exists because the cursor displacement 
used in the visual display has a consistent, constant mapping 
to the mouse displacement used in the force "display.** 
However, if ballistics are used to map physical mouse 
motion to displayed cursor motion, the motion of the cursor 
varies depending upon mouse velocity, causing a potential 
conflict: the stretch displayed visually is based on the 
variable mapping adjusted by the ballistic algorithm, while 
the stretch force is based on pure mouse motion. This 
conflict becomes a problem when, for example, the user 
stretches a line very quickly in one direction from a starling 



position (both a mouse starting position and a screen starting 
position), changes to the opposite direction, and unstretches 
the line very slowly toward the starling position. The motion 
in the first direction has a large scaling of mouse motion to 
cursor motion, while the motion in the second direction has 
a smaU scaling of mouse motion to cursor motion. The user 
may move the mouse in its physical workspace exactly the 
same distance in both directions (returning to the mouse 
starting position), but on the screen the line might stretch 
very far (when moving fast), but then come back only a 
small distance (when moving slow). Thus, visually, the 
cursor did not return to the screen starting position. 

[0012] Using a traditional mouse, this frequently occurs 
and is not a problem. However, on a force feedback mouse 
where force display is based on mouse motion and visual 
display is based on cursor motion, a disconcerting 
dichotomy is noticed by the user. Since the feel of the stretch 
is based on mouse motion, if the user stretches in one 
direction and returns to the starting position, the user feels 
stretching the line a given displacement and then imstretch- 
ing the line that same displacement, with an end result of no 
stretch. But, on the screen where the cursor mapping is based 
on velocity, the user would see the line stretching far in one 
direction and then unstretching only a small amount in the 
opposite direction so that the cursor is not yet back to the 
starling position. The user would visually expect to still feel 
some stretched tension, but no such tension exists since the 
mouse is back al the starting position in its own workspace. 
Thus a problem is evident in force feedback mouse-type 
devices: ballistics are needed to allow dexterous cursor 
control, yet ballistics distorts the seeing- feeling relationship. 

[0013] Thus, adjusting the mapping between physical 
mouse motion and displayed cursor motion makes sense for 
graphical display, but does not make sense for force feed- 
back where physical realism is critical to effective sensation 
generation. Therefore, for force feedback mouse systems, it 
would be preferred to eliminate mouse ballistics. Unfortu- 
nately, such a force feedback mouse would not be optimized 
for both fine positioning and coarse motion, as is true of 
traditional mice. 

[0014] In addition, mouse ballistics causes another prob- 
lem that causes diflScuhy in force feedback mouse imple- 
mentation. As described above, moving the mouse in one 
direction quickly and then moving it back in the other 
direction slowly creates a situation where the mouse hard- 
ware has returned to its starting position but the cursor may 
be far away from its starting position. This illustrates that the 
frame of the cursor and the frame of the mouse have shifted 
or become offset. 

[0015] If this offset becomes too large, the user may not be 
able to reach some parts of the screen within the range of 
motion of the mouse. In a typical mouse, the offset is 
corrected through a process called "indexing." Indexing is 
achieved in a typical mouse by lifting the mouse off the table 
and repositioning it after the mouse has hit a limit, while the 
cursor remains fixed in position. This brings the mouse and 
the cursor frames back to a smaller, more comfortable offset. 
A force feedback mouse may have a limited workspace due 
to cost constraints and may not be able to be lifted off the 
table and repositioned. In addition, the mouse hilling a 
physical hmit lo its woricspace is disconcerting for a user 
expecting realistic force feedback. Thus, traditional indexing 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



3 



Jan. 10, 2002 



may not be practical. However, since ballistics needs index- 
ing to restore the frame offsets, and since ballistics and 
indexing are both traditional mouse techniques that conflict 
wixh typical force feedback functionality, a solution is 
needed that reconciles both the ballistics and the indexing 
problem in force feedback hardware, 

SUMMARY OF THE I^A^NnON 

[0016] The present invention is directed to a force feed- 
back interface which allows enhanced cursor control and 
does not compromise the fidelity of force feedback. Various 
embodiments are presented which distinguish when a con- 
flict between cursor control and force feedback occurs and 
compensates for such conflict, as well as compensating for 
any indexing problems occurring due to offsets in mouse and 
cursor movement frames. 

[0017] More specifically, a method of the present inven- 
tion provides enhanced cursor control using a force feedback 
interface device coupled to host computer. The host com- 
puter displays a cursor within a graphical environment, such 
as a GUI, on a display device. A position of a user-grasped 
manipulandum, such as a mouse, in a device workspace is 
read as a reference position. A cursor position is reported to 
the host computer derived from the reference position, and 
the host computer displays the cursor within the graphical 
environment at a position corresponding to the cursor posi- 
tion. It is determined whether the cursor interacts with the 
graphical environment as to cause a force to be output on the 
manipulandum, and, if so, a force is output on the manipu- 
landum. At least one of the reported cursor position and the 
output force allows the user of the force feedback interface 
device to finely position the cursor within the graphical 
environment and coarsely move the cursor in the graphical 
environment without causing a distortion in the output 
forces as expected to be experienced by the user. 

[0018] A number of embodiments are particularly 
described. In some embodiments, the cursor position 
reported to the host computer is the reference position 
modified or scaled to allow enhanced cursor control. For 
example, the cursor position can be a ballistic position that 
is the reference position modified by a ballistics algorithm 
such that cursor position is mapped to manipulandum posi- 
tion based on a scaling derived from a velocity of the 
manipulandum and allows enhanced cursor control. In other 
embodiments, other types of variable scaling or variable 
mapping can be used, such as a predictive scaling method 
that scales the cursor position based on whether a fine 
positioning mode is entered based on other criteria. In such 
ballistics or variable scaling/mapping embodiments, the 
realism of output forces is maintained by determining 
mouse-based forces based on the reference data (position 
and motion of the mouse) rather than the cursor position 
(ballistic) data. Preferably, a local microprocessor keeps 
track of both reference data (local frame) and the ballistic 
data (display frame) and uses data from each as appropriate. 
In one embodiment, the cursor position is a ballistic position 
except for when visual spring forces are output, which 
would cause a conflict in the cursor position and the expe- 
rienced force. Thus, the cursor position sent to the host is the 
reference position modified by a constant mapping when the 
output force is a visual spring force instead of being modi- 
fied by ballistics. In a different, preferred embodiment, most 
motion -based output forces are again based on reference 



data and the cursor position based on ballistic data, but 
particular forces that cause the conflict such as the visual 
spring force are determined using the ballistic data. 

[0019] In other embodiments, enhanced cursor control is 
provided by oulputting assistive forces. In one embodiment, 
a resistive force dedicated for enhanced cursor control, such 
as a damping for friction force, is output to slow movement 
of the manipulandum. The resistive force has a magnitude 
inversely based on a velocity of the manipulandum in the 
device workspace to allow the mouse to be slowed down for 
fine positioning tasks and freed or unencumbered for coarse 
positioning tasks. In a different embodiment, a detent force 
is associated with targets in the GUI for guiding the manipu- 
landum to a particular position and thereby guiding said 
cursor to a corresponding position in the graphical environ- 
ment. In another embodiment, the detent forces are provided 
as a field of detents arranged in a predetermined spacing 
over a predetermined area surrounding the cursor, which are 
provided when the manipulandum is under a predetermined 
velocity and thus likely to need the detents for fine posi- 
tioning of the cursor. In a different embodiment, obstruction 
forces simulating surfaces are arranged to assist the user to 
controlling the cursor in fine positioning tasks. 

[0020] An indexing feature of the present invention allows 
control over the cursor by the mouse when an ofiEset exists 
between the position of the mouse in its workspace (local 
frame) and the position of the cursor on the display screen 
(display frame). The mouse device is coupled to a host 
computer that displays graphical objects in a graphical 
environment on a display screen and includes a mouse 
moveable in a mouse workspace. The cursor is moved in a 
screen area based on the movement of said mouse. ITie 
mouse is determined whether it is within a predetermined 
distance to a physical limit of the mouse workspace, the 
predetermined distance being defined by a region next to 
said physical limit. A location of the mouse in the region is 
determined, and the location is used to provide control of 
movement of the cursor toward the screen edge correspond- 
ing to the physical limit. A cursor position is reported to the 
host computer allowing control of the cursor to the edge of 
said screen area despite the offset between local and display 
frames. A force feedback mouse device that provides an 
indexing function is similar to the method. 

[0021] In one embodiment of the indexing feature, the 
distance of penetration of the mouse into the region is 
sensed, and a resistive force is output on the mouse resisting 
the movement into the region. For example, the force may 
be a resistive spring force having a magnitude based on the 
distance of the mouse past the region border. The penetration 
distance is used to provide control of movement of the 
cursor toward a screen limit of the display frame corre- 
sponding to the physical limit according to a isometric 
control paradigm. The speed of the cursor may be based on 
the distance of the mouse past the region border (e.g. the 
compression of the simulated spring). In a different embodi- 
ment of an indexing feature, a first distance between a 
current position of the mouse and the closest physical limit 
of the workspace is determined, and a second distance 
between a current position of the cursor and the edge to the 
display screen corresponding to that physical limit is deter- 
mined. A ratio between the first and second distances is used 
to determine a scaling for determining cursor position, thus 
allowing the cursor to be positioned to the edge of said 



12/08/2003, EAST Version: 1.4.1 



us 2002/0003528 Al Jan. 10, 2002 



screen area when (or before) the mouse reaches the physical 
limit to the workspace. In ooe embodiment, this scaling is 
performed only when the mouse is within a predetermined 
region adjacent to a physical limit of the mouse workspace. 

[0022] An interface device of the present invention pro- 
viding enhanced cursor control over a cursor includes a user 
manipulatable physical object contacted by a user and 
movable in physical space, a sensor that detects movement 
of the physical object in physical space and an actuator that 
applies output forces on the physical object. A local micro- 
processor is preferably included for determining and report- 
ing the cursor position to the host, determining and output- 
ting forces, and determining indexing functions similarly as 
in the above embodiments. 

[0023] llie methods and apparatus of the present invention 
advantageously provides enhanced conUrol over a cursor in 
a graphical environment while not compromising the fidelity 
or expected feel of force feedback sensations based on 
motion of the mouse or other user object. This allows a user 
to perform fine positioning and coarse motion of the cursor 
as desired and still experience forces as expected based on 
interactions of the cursor in the graphical environment. In 
addition, the indexing features of the present invention allow 
the user to control the cursor even when a large offset exists 
between the mouse and cursor positions in their respective 
frames, allows the user to reduce this ofl&et, and substan- 
tially reduces the user's undesired experience of any hard, 
physical stops when the motise reaches a physical limit. 

[0024] These and other advantages of the present inven- 
tion will become apparent to those skilled in the art upon a 
reading of the following specification of the invention and a 
study of the several figures of the drawing. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0025] FIG. 1 is a perspective view of one embodiment of 
a mouse interface system suitable for use with the present 
invention; 

[0026] FIG, 2 is a perspective view of an embodiment of 
a mechanism suitable for the interface system of FIG. 1; 

[0027] FIGS. 3«-6 are top plan views of the mechanism of 
FIG. 2; 

[0028] FIG. 4 is a block diagram of the system of FIG. 1 
for controlling a force feedback interface device of the 
present invention; 

[0029] FIG. 5 is a diagrammatic illustration of the local 
frame and host frame referenced in the present invention; 

[0030] FIG. 6 is a flow diagram illustrating a first embodi- 
ment of a method of the present invention for providing 
enhanced cursor control without compromising force feed- 
back; 

[0031] FIGS, la-lf are diagrammatic illustrations of the 
dichotomy between a display frame and a local frame when 
oulputting a visual spring force; 

[0032] FIG. 8 is a flow diagram illustrating a second 
embodiment of a method of the present invention for pro- 
viding enhanced cursor control without compromising force 
feedback; 



[0033] FIG. 9 is a flow diagram illusu-ating a third 
embodiment of a method of the present invention for pro- 
viding enhanced cursor control without compromising force 
feedback including a first embodiment of an indexing fea- 
ture of the present invention; 

[0034] Figures lOa-lOc are diagrammatic illustrations 
demonstrating the isometric indexing function of the present 
invention; 

[0035] FIG.' 11 is a flow diagram illustrating a fourth 
embodiment of a method of the present invention for pro- 
viding enhanced cursor control without compromising force 
feedback including a second embodiment of an indexing 
feature of the present invention; 

[0036] FIG. 12 is a flow diagram illustrating a fifth 
embodiment of a method of the present invention for pro- 
viding enhanced cursor control without compromising force 
feedback; 

[0037] FIG. 13 is a flow diagram illustrating a sixth 
embodiment of a method of the present invention for pro- 
viding enhanced cursor control without compromising force 
feedback; and 

[0038] FIG. 14 is a flow diagram illustrating a seventh 
embodiment of a method of the present invention for pro- 
viding enhanced cursor control without compromising force 
feedback. 

DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS 

[0039] FIG. 1 is a perspective view of a force feedback 
mouse interface system 10 of the present invention capable 
of providing input to a host computer based on the user's 
manipulation of the mouse and capable of providing force 
feedback to the user of the mouse system based on events 
occurring in a program implemented by the host computer. 
Mouse system 10 includes a mouse or "puck"12, an inter- 
face 14, and a host computer 18. It should be noted that the 
tenn "mouse" as used herein, indicates an object 12 gener- 
ally shaped to be grasped or contacted from above and 
moved within a substantially planar workspace (and addi- 
tional degrees of freedom if available). Typically, a mouse is 
a smooth or angular shaped compact unit that snugly fits 
under a user's hand, fingers, and/or palm, but may be shaped 
otherwise in other embodiments. 

[0040] Mouse 12 is an object that is preferably grasped or 
gripped and manipulated by a user. By "grasp," it is meant 
that users may releasably engage a portion of the object in 
some fashion, such as by hand, with their fingertips, etc. In 
the described embodiment, mouse 12 is shaped so that a 
user's fingers or hand may comfortably grasp the object and 
move it in the provided degrees of fi-eedom in physical 
space; an example of a user's hand is shown as dashed line 
16. For example, a user can move mouse 12 to correspond- 
ingly move a computer generated graphical object, such as 
a cursor or other image, in a graphical environment provided 
by computer 18. The available degrees of fi-eedom in which 
mouse 12 can be moved are determined from the interface 
14, described below. In addition, mouse 12 preferably 
includes one or more buttons 15 to allow the user to provide 
additional commands to the computer system. 

[0041] It will be appreciated thai a great number of other 
types of user manipulable objects ("user objects" or "physi- 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



5 



Jan. 10, 2002 



cal objects") can be used with the method and apparatus of 
the present invention in place of or in addition to mouse 12. 
For example, such objects may include a sphere such as a 
track ball, a puck, a joystick, cubical- or other-shaped hand 
grips, a receptacle for receiving a finger or a stylus, a flat 
planar surface like a plastic card having a rubberized, 
contoured, and/or bumpy surface, or other objects, 

[0042] Interface 14 interfaces mechanical and electrical 
input and output between the mouse 12 and host computer 
18 implementing the application program, such as a GUI, 
simulation or game environment. Interface 14 provides 
multiple degrees of freedom to mouse 12; in the preferred 
embodiment, two planar degrees of freedom are provided to 
the mouse, as shown by arrows 22. In other embodiments, 
greater or fewer degrees of freedom can be provided, as well 
as rotary degrees of freedom. For many applications, mouse 
12 need only be moved in a very small workspace area, 
shown as dashed line 24 in FIG. 1 as an example. 

[0043] In a preferred embodiment, the user manipulates 
mouse 12 in a planar workspace and the position of mouse 
12 is translated into a form suitable for interpretation by 
position sensors of the interface 14. The sensors track the 
movement of the mouse 12 in planar space and provide 
suitable electronic signals to an electronic portion of inter- 
face 14. The interface 14 provides position information to 
host computer 18, which the host uses, for example, to 
display a cursor or other user-controlled graphical object. In 
addition, host computer 18 and/or interface 14 provide force 
-feedback signals to actuators coupled to interface 14, and the 
actuators generate forces on members of the mechanical 
portion of the interface 14 to provide forces on mouse 12 in 
provided or desired degrees of freedom. The user experi- 
ences the forces generated on the mouse 12 as realistic 
simulations of force sensations such as jolts, springs, tex- 
tures, "barrier" forces, and the like. 

[0044] The electronic portion of interface 14 may couple 
the mechanical portion of the interface to the host computer 
18. The electronic portion is preferably included within the 
housing 26 of the interface 14 or, alternatively, the electronic 
portion may be included in host computer 18 or as a separate 
unit with its own housing. More particularly, interface 14 
preferably includes a local microprocessor distinct and sepa- 
rate from any microprocessors in the host computer 18 to 
control force feedback on mouse 12 independently of the 
host computer, as well as sensor and actuator interfaces. A 
suitable embodiment of the electrical portion of interface 14 
is described in detail with reference to FIG. 4. 

[0045] The interface 14 can be coupled to the computer 18 
by a bus 17, which communicates signals between interface 
14 and computer 18 and also, in the preferred embodiment, 
provides power to the interface 14 (e.g. when bus 17 
includes a USB interface). In other embodiments, signals 
can be sent between interface 14 and computer 18 by 
wireless transmission/reception. In preferred embodiments 
of the present invention, the interface 14 serves as an 
input/output (I/O) device for the computer 18. The interface 
14 can also receive inputs from other input devices or 
controls that are associated mouse system 10 and can relay 
those inputs to computer 18. For example, commands sent 
by the user activating a button 15 on mouse 12 can be 
relayed to computer 18 by interface 14 to implement a 
command or cause the computer 18 to output a command to 
the interface 14. 



[0046] Host computer 18 is preferably a personal com- 
puter or workstation, such as an EBM-PC compatible com- 
puter or Macintosh personal computer, or a SUN or Silicon 
Graphics workstation. For example, the computer 18 can 
operate under the Windows'™ or MS-DOS operating system 
in conformance with an IBM PC AT standard. Alternatively, 
host computer system 18 can be one of a variety of home 
video game systems commonly connected to a television set, 
such as systems available from Nintendo, Sega, or Sony. In 
other embodiments, home computer system 18 can be a "set 
top box" which can be used, for example, to provide 
interactive television functions to users, or a "network-" or 
"Internet-computer" which allows users to interact with a 
local or global network using standard connections and 
protocols such as used for the Internet and World Wide Web. 
Host computer preferably includes a host microprocessor, 
random access memory (RAM), read only memory (ROM), 
input/output (I/O) circuitry, and other components of com- 
puters well-known to those skilled in the art. 

[0047] Host computer 18 preferably implements a host 
application program with which a user is interacting via 
mouse 12 and other peripherals, if appropriate, and which 
can include force feedback functionality. For example, the 
host application program can be a simulation, video game, 
Web page or browser that implements HTML, VRML, or 
other instructions, scientific analysis program, virtual reality 
training program or application, or other application pro- 
gram that utilizes input of mouse 12 and outputs force 
feedback commands to the mouse 12. Herein, for simplicity, 
operating systems such as Windows™, MS-DOS, MacOS, 
Unix, etc. are also referred to as "application programs." In 
one preferred embodiment, an application program utilizes 
a graphical user interface (GUI) to present options to a user 
and receive input from the user. Herein, computer 18 may be 
referred as displaying "graphical objects" or "computer 
objects." These objects arc not physical objects, but are 
logical software unit collections of data and/or procedures 
that may be displayed as images by computer 18 on display 
screen 20, as is well known to those skilled in the art. A 
displayed cursor or a simulated cockpit of an aircraft might 
be considered a graphical object. The host application pro- 
gram checks for input signals received from the electronics 
and sensors of interface 14, and outputs force values and/or 
commands to cause the output of forces on mouse 12. 
Suitable software drivers which interface such simulation 
software with computer input/output (I/O) devices are avail- 
able from Immersion Human Interface Corporation of San 
Jose, Calif. 

[0048] Display device 20 can be included in host computer 
18 and can be a standard display screen (LCD, CRT, etc.), 
3-D goggles, or any other visual output device. Typically, the 
host application provides images to be displayed on display 
device 20 and/or other feedback, such as auditory signals. 
For example, display screen 20 can display images from a 
GUI. Images describing a moving, first person point of view 
can be displayed, as in a virtual reality game. Or, images 
describing a third -person perspective of objects, back- 
grounds, etc. can be displayed. Alternatively, images from a 
simulation can be displayed. Images may be displayed 
and/or modified on display device 20 in response to user 
manipulations of mouse 12. 

[0049] There are two primary "control paradigms" of 
operation for mouse system 10: position control and rate 



12/08/2003. EAST Version: 1.4.1 



us 2002/0003528 Al 



6 



Jan. 10, 2002 



control. Position control is the more typical control para- 
digm for mouse and similar controllers, and refers to a 
mapping of mouse 12 in which displacement of the mouse 
in physical space directly dictates displacement of a graphi- 
cal object. The mapping can have an arbitrary scale factor, 
but the fundamental relation between mouse displacements 
and graphical object displacements should be present. Under 
a position control mapping, the computer object does not 
move unless the user object is in motion. Position control is 
a popular mapping for applications such as graphical user 
interfaces (GUI's) or medical procedure simulations. Posi- 
tion control force feedback roughly corresponds to forces 
which would be perceived directly by the user, i.e., they are 
"user-centric" forces. 

[0050] As shown in FIG. 1, a"display frame'*28 is pro- 
vided with the display screen 20 for defining the area of 
movement of a cursor in graphical environment. This frame 
can also be considered a "host frame", although the interface 
14 may reference it as well. In contrast, the mouse 12 has a 
"local frame"30 allowed by the workspace in which the 
mouse 12 is moved. In a position control paradigm, the 
position (or change in position) of a user-controlled graphi- 
cal object, such as a cursor, in display frame 30 corresponds 
to a position (or change in position) of the mouse 12 in the 
local frame 28. 

[0051] Rate control is also used as a control paradigm. 
This refers to a mapping in which the displacement of the 
mouse 12 along one or more provided degrees of freedom is 
abstractly mapped to motion of a computer-simulated object 
under control. There is not a direct physical mapping 
between physical object (mouse) motion and computer 
object motion. Thus, most rate control paradigms allow the 
user object can be held steady at a given position but the 
controlled computer object is in motion at a commanded or 
given velocity, in contrast to the position control paradigm 
that only allows the controlled computer object to be in 
motion if the user object is in motion. 

[0052] 'Ilie mouse interface system 10 is useful for both 
position control ("isotonic") tasks and rate control ("isomet- 
ric") tasks. For example, as a traditional mouse, the position 
of mouse 12 in the workspace 24 can be directly mapped to 
a position of a cursor on display screen 20 in a position 
control paradigm. Alternatively, the displacement of mouse 
12 in a particular direction against an opposing output force 
can command rate control tasks in an isometric mode, as 
described with reference to the indexing feature of FIG, 13. 
Another implementation that provides both isotonic and 
isometric functionality for a force feedback controller and 
which is suitable for the interface device of the present 
invention is described in patent application Sen No. 08/756, 
745, incorporated by reference herein. 

[0053] Mouse 12 can be used, for example, to control a 
computer-generated graphical object such as a cursor dis- 
played in a graphical computer environment,' such as a GUI. 
The user can move the mouse in 2D planar workspace to 
move the cursor to graphical objects in the GUI or perform 
other tasks. In other graphical environments, such as a 
virtual reality video game, a user can be controlling a 
computer player or vehicle in the virtual environment by 
manipulating the mouse 12. The computer system tracks the 
position of the mouse with sensors as the user moves it. The 
computer system may also provide force feedback com- 



mands to the mouse, for example, when the user moves the 
graphical object against a generated surface such as an edge 
of a window, a virtual wall, etc. It thus appears and feels to 
the user that the mouse and the graphical object arc con- 
tacting real surfaces. 

[0054] FIG. 2 is a perspective view of one embodiment of 
mouse system 10 with the cover portion of housing 26 
removed, showing the mechanical portion of interface 14 for 
providing mechanical input and output in accordance with 
the present invention. The mouse device 10 of FIG. 2 is 
described in greater detail in co-pending patent application 

08/ , filed Jun. 24, 1997, entitled, "Force Feedback 

Mouse Interface", and incorporated by reference herein in its 
entirety. 

[0055] Interface 14 includes a mouse or other user 
manipulatable object 12, a mechanical linkage 40, and a 
transducer system 41. A base 42 is provided to support the 
mechanical linkage 40 and transducer system 41 on 
grounded surface 34. Mechanical linkage 40 provides sup- 
port for mouse 12 and couples the mouse to a groimded 
sxirfacc 34, such as a tabletop or other support. Linkage -40 
is, in the described embodiment, a 5-member (or "5-bar") 
linkage including a ground member 42, a first base member 
44 coupled to ground member 42, a second base member 48 
coupled to ground member 42, a link member 46 coupled to 
base member 44, and an object member 50 coupled to link 
member 46, base member 48 and to mouse 12. Fewer or 
greater numbers of members in the linkage can be provided 
in alternate embodiments. 

[0056] Ground member 42 of the linkage 40 is a base for 
the support of the linkage and is coupled to or resting on a 
ground surface 34. The members of linkage 40 are rotatably 
coupled to one another through the use of rotatable pivots or 
bearing assemblies ("bearings") having one or more bear- 
ings. Base member 44 is rotatably coupled to ground mem- 
ber 42 by a grounded bearing 52 and can rotate about an axis 
A. Link member 46 is rotatably coupled to base member 44 
by bearing 54 and can rotate about a floating axis fi, and base 
member 48 Is rotatably coupled to ground member 42 by 
bearing 52 and can rotate about axis A. Object member 50 
is rotatably coupled to base member 48 by bearing 56 and 
can rotate about floating axis C, and object member 50 is 
also rotatably coupled to link member 46 by bearing 58 such 
that object member 50 and link member 46 may rotate 
relative to each other about floating axis D. Linkage 40 is 
formed as a five- member closed-loop chain arranged such 
that the members can rotate about their respective axes to 
provide mouse 12 with two degrees of freedom, i.e., mouse 
12 can be moved within a planar workspace defined by the 
x-y plane, which is defined by the x- and y-axes as shown 
in FIG. 2. Mouse 12 in the preferred embodiment is coupled 
to object member 50 by a rotary bearing 60 so that the mouse 
may rotate about floating axis E and allow the user some 
flexible movement in the planar workspace. 

[0057] Transducer system 41 is used to sense the position 
of mouse 12 in its workspace and to generate forces on the^ 
mouse 12. Transducer system 41 preferably includes sensors 
62 and actuators 64. The sensors 62 collectively sense the 
movement of the mouse 12 in the provided degrees of 
freedom and send appropriate signals to the electronic 
portion of interface 14. Sensor 62a senses movement of link 
member 48 about axis A, and sensor 62fo senses movement 



12/08/2003, EAST Version: 1.4.1 



I 



us 2002/0003528 Al Jan. 10, 2002 

7 



of base member 44 about axis A. These sensed positions 
about axis A allow the determinalion of the position of 
mouse 12 using known constants such as the lengths of the 
members of linkage 40 and using well-known coordinate 
transformations. 

[0058] Sensors 62 are, in the described embodiment, 
grounded optical encoders that sense the intermittent block- 
age of an emitted beam. A grounded emitter portion 70 emits 
a beam which is detected across a gap by a grounded 
detector 72. A moving encoder disk or arc 74 is provided at 
the end of member 4S which blocks the beam in predeter- 
mined spatial increments and allows a processor to deter- 
mine the position of the arc 74 and thus the member 48 by 
counting the spatial increments. Also, a velocity of member 
48 based on the speed of passing encoder marks can also be 
determined. 

[0059] Transducer system 41 also preferably includes 
^actuators 64 to transmit forces to mouse 12 in space, i.e., in 
two (or more) degrees of freedom of the user object. The 
housing of a grounded portion of actuator 64b is rigidly 
coupled to ground member 42 and a moving portion of 
actuator 64^> (preferably a coil) is integrated into the base 
member 44. llie actuator transmits rotational forces to base 
member 44 about axis A. The housing of the grounded 
portion of actuator 64fl is rigidly coupled to ground member 
42 through the grounded housing of actuator 64i», and a 
moving portion (preferably a wire coil) of actuator 64a is 
integrated into base member 48. Actuator 64a transmits 
rotational forces to link member 48 about axis A. The 
combination of these rotational forces about axis A allows 
forces to be transmitted to mouse 12 in all directions in the 
planar workspace provided by linkage 40 through the rota- 
tional interaction of the members of linkage 40. 

[0060] In the preferred embodiment, actuators 64 arc 
electromagnetic voice coil actuators which provide force 
through the interaction of a current in a magnetic field. The 
magnetic fields from magnets of the actuators interact with 
a magnetic field produced from the wire coil when current 
is flowed in the coil, thereby producing forces on appropriate 
members. The magnitude or strength of the force is depen- 
dent on the magnitude of the current that is applied tolhe 
coil, the number of loops in the coil, and the magnetic field 
strength of the magnets. The direction of the force depends 
on the direction of the current in the coil. Forces in the x- and 
y-directions of mouse 10 are thus produced. The operation 
of voice coil actuators is described in greater detail in 
copending patent application Ser. No, 08/560,091, incorpo- 
rated by reference herein. In other embodiments, other types 
of actuators can be used, both active and passive, such as DC 
motors, pneumatic motors, passive friction brakes, passive 
fluid-controlled brakes, etc. Voice coil actuators can also be 
used as sensors to sense the velocity (and thus position and 
acceleration) of the members 44 and 48 about axis A. 

[0061] In an alternate embodiment, the mechanism 14 can 
be used for a 3-D interface device that allows a user to move 
a user object 12 in three dimensions rather than the 2-D 
planar workspace disclosed. For example, in one embodi- 
ment, the entire mechanism 14 can be made to rotate about 
a grounded axis, such as axis H extending through the 
magnet assemblies 88. 

[0062] As shown in FIG. 3a, a workspace guide opening 
76 is provided in ground member 42 to limit the movement 



of mouse 12 in the x-y plane and thus defines the physical 
workspace of the mouse 12. Guide opening 76 is a shallow 
opening in the grotmd member 42 having sides which block 
movement of the mouse 12 beyond specified limits. A guide 
pin 78 is coupled to the bearing 60 at axis E and extends 
down into the guide opening 76. Pin 78 contacts one or more 
sides of the opening 76 when the mouse is moved to a limit 
in a particular direction. As shown, guide opening 76 has 
relatively small dimensions, allowing the mouse a work- 
space of approximately 0.9" by 0.9" in the described 
embodiment. This is typically adequate workspace for the 
user to move the mouse and control a graphical object such 
as a cursor on a display screen. In other embcidiments, 
differently-sized guide openings can be provided for differ- 
ently-sized workspaces, or other types of stops or guides can 
be used to prevent movement past predetermined limits. The 
guide opening 76 is shown as square shaped, but it can be 
rectangular in other embodiments; for example, the dimen- 
sions of opening 76 can be made the same aspect ratio as the 
displayed area of display device 20. FIG. 3a shows guide 
pin 78 approximately in the center of the guide opening 76. 

[0063] In FIG. 36, the mouse 12 (not shown) and axis E 
have been moved in the x-y plane of the workspace of the 
mouse. The movement of the mouse has been limited by the 
guide opening 76, where guide pin 78 has engaged the 
sidewall of the upper-left comer area of guide opening 76 
and stops any further movement in the forward y-direction. 

[0064] FIG. 4 is a block diagram illustrating the electronic 
portion of interface 14 and host computer 18 suitable for use 
with the present invention. Mouse interface system 10 
includes a host computer 18, electronic interface 100, 
mechanical apparatus 102, and mouse or other user object 
12, Electronic interface 100, mechanicar apparatus 102, and 
mouse 12 can also collectively be considered a "force 
feedback interface device"104 that is coupled to the host 
computer. A similar system is described in detail in co- 
pending patent application Sen No. 08/566,282, which is 
hereby incorporated by reference herein. 

[0065] As explained with reference to FIG. 1, computer 
18 is preferably a personal computer, workstation, video 
game console, or other computing or display device. Host 
computer system 18 commonly includes a host micropro- 
cessor 108, random access memory (RAM) 110, read-only 
memory (ROM) 112, input/output (I/O) electronics 114, a 
clock 116, a display device 20, and an audio output device 
118, Host microprocessor 108 can include a variety of 
available microprocessors from Intel, AMD, Motorola, or 
other manufacturers, and can be single microprocessor chip, 
or multiple primary and/or co-processors. Microprocessor 
108 preferably retrieves and stores instructions and other 
necessary data from RAM 110 and ROM 112 as is well 
known to those skilled in the art. In the described embodi- 
ment, host computer system 18 can receive sensor data or a 
sensor signal via a bus 120 from sensors of system 10 and 
other information. Microprocessor 108 can receive data 
from bus 120 using I/O electronics 114, and can use I/O 
electronics to conu-ol other peripheral devices. Host com- 
puter system 18 can also output commands to interface 
device 104 via bus 120 to cause force feedback for the 
interface system 10. 

[0066] Clock 116 is a standard clock crystal or equivalent 
component used by host computer 18 to provide timing to 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



8 



Jan. 10, 2002 



electrical signals used by host microprocessor 108 and other 
components of the computer system 18. Clock 116 is 
accessed by host computer 18 in the control process of the 
present invention to provide timing information that may be 
necessary in determining force or position, e.g., calculating 
a velocity or acceleration from position values, 

[0067] Display device 20 is described with reference to 
FIG. 1. Audio output device 118, such as speakers, can be 
coupled to host microprocessor 108 via amplifiers, filters, 
and other circuitry well known to those skilled in the art. 
Host processor 108 outputs signals to speakers 118 to 
provide sound output to the user when an "audio event" 
occurs during the implementation of the host application 
program. Other types of peripherals can also be coupled to 
host processor 108, such as storage devices (hard disk drive, 
CD ROM drive, floppy disk drive, etc.), printers, and other 
input and output devices. 

[0068] Electronic interface 100 is coupled to host com- 
puter system 18 by a bi-directional bus 120. The bi-direc- 
tional bus sends signals in either direction between host 
computer system 18 and the interface device 104. Bus 120 
can be a serial interface bus providing data according to a 
serial communication protocol, a parallel bus using a par- 
allel protocol, or other types of buses. An interface port of 
host computer system 18, such as an RS232 serial interface 
port, connects bus 120 to host computer system 18. In 
another embodiment, an additional bus 122 can be included 
to communicate between host computer system 18 and 
interface device 13. Bus 122 can be coupled to a second port 
of the host computer system, such as a "game port", such 
that two buses 120 and 122 are used simultaneously to 
provide an increased data bandwidth. One preferred serial 
interface bus used in the present invention is the Universal 
Serial Bus (USB). The USB standard provides a relatively 
high speed serial interface that can provide force feedback 
signals in the present invention with a high degree of 
realism. USB can also source power to drive actuators 64 
and other devices of the present invention, and can provide 
timing data that is encoded along with differential data. 

[0069] Electronic interface 100 includes a local micropro- 
cessor 130, local clock 132, local memory 134, sensor 
interface 136, and actuator interface 138. Interface 100 may 
also include additional electronic components for commu- 
nicating via standard protocols on buses 120 and 122. 

[0070] In various embodiments, electronic interface 100 
can be included in mechanical apparatus 102, in host com- 
puter 18, or in its own separate housing. 

[0071] Local microprocessor 130 preferably coupled to 
bus 120 and may be closely linked to mechanical apparatus 
102 to allow quick communication with other components 
of the interface device. Processor 130 is considered "local" 
to interface device 104, where "local" herein refers to 
processor 130 being a separate microprocessor firom any 
processors 108 in host computer 18. "Local" also preferably 
refers to processor 130 being dedicated to force feedback 
and sensor I/O of the interface system 10, and being closely 
coupled to sensors and actuators of the mechanical apparatus 
102, such as within the housing of or in a housing coupled 
closely to apparatus 102. Microprocessor 130 can be pro- 
vided with software instructions to wait for commands or 
requests from computer host 18, parse/decode the command 
or request, and handle/control input and output signals 



according to the command or request. In addition, processor 
130 preferably operates independently of host computer 18 
by reading sensor signals and calculating appropriate forces 
from those sensor signals, time signals, and force processes 
selected in accordance with a host command, and output 
appropriate control signals to the actuators. Suitable micro- 
processors for use as local microprocessor 200 include the 
MC68HC711E9 by Motorola, the PIC16C74 by Microchip, 
and the 80930 from Intel, for example. Microprocessor 130 
can include one microprocessor chip, or multiple processors 
and/or co-processor chips, and/or digital signal processor 
(DSP) functionality. 

[0072] For example, in one host -controlled embodiment 
that utilizes microprocessor 130, host computer 18 can 
provide low-level force comnaands over bus 120, which 
microprocessor 130 directly transmits to the actuators. In a 
different local control embodiment, host computer system 
18 provides high level supervisory commands to micropro- 
cessor 130 over bus 120, and microprocessor 130 manages 
low level force control loops to sensors and acmators in 
accordance with the high level commands and indepen- 
dently of the host computer 18. In the local control embodi- 
ment, the microprocessor 130 can process inputted sensor 
signals to deteriaine appropriate output actuator signals by 
following the instructions of a "force process" that may be 
stored in local memory and includes calculation instructions, 
formulas, force magnitudes, or other data. The force process 
can command distinct force sensations, such as vibrations, 
textures, jolts, or even simulated interactions between dis- 
played objects. Force feedback used in graphical environ- 
ments is described in greater detail in co -pending patent 
application Ser. Nos. 08/571,606, 08/756,745, and 

08/ , entitled, "Graphical Click Surfaces for Force 

Feedback Applications", by Rosenberg et al, filed Jun. 18, 
1997, all of which are incorporated by reference herein. 

[0073] For example, a rigid surface is generated on com- 
puter screen 20 and a computer object (e.g., cursor) con- 
trolled by the user collides with the surface. In a preferred 
embodiment, high-level host commands can be used to 
provide the various forces associated with the rigid surface. 

[0074] A local control mode using microprocessor 130 can 
be helpful in increasing the response time for forces applied 
to the user object, which is essential in creating realistic aiid 
accurate force feedback. 

[0075] For example, it is preferable that host computer 18 
send a "spatial representation" to microprocessor 130, which 
is data describing the locations of some or all the graphical 
objects displayed in a GUI or other graphical environment 
which are associated with forces and the types/characteris- 
tics of these graphical objects. The microprocessor can store 
such a spatial representation in memory 134, and thus will 
be able to determine interactions between the user object and 
graphical objects (such as the rigid surface) independently of 
the host computer. In addition, the microprocessor 130 can 
be provided with the necessary instructions or data to check 
sensor readings, determine cursor and target positions, and 
determine output forces independently of host computer 18. 
The host can implement program functions (such as dis- 
playing images) when appropriate, and synchronization 
commands can be communicated between processor 130 
and host 18 to correlate the microprocessor and host pro- 
cesses. Also, memory 134 can store predetermined force 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



sensations for microprocessor 130 that are to be associated 
vdih particular types of graphical objects. Alternatively, the 
computer 18 can directly send force feedback signals to the 
interface 14 to generate forces on mouse 12. 

[0076] Sensor signals used by microprocessor 130 are also 
reported to host computer system 18, which updates a host 
application program and outputs force control signals as 
appropriate. For example, if the user moves mouse 12, the 
computer system 18 receives cursor position signals indi- 
cating this movement and can move a displayed cursor in 
response. These embodiments are described in greater detail 
in co-pending applications Ser. Nos. 08/534,791 and 08/566, 
282. In an alternate embodiment, no local microprocessor 
130 is included in interface system 10, and host computer 18 
directly controls and processes all signals to and from the 
interface 100 and mechanical interface 102. 

[0077] A local clock 132 can be coupled to the micropro- 
cessor 130 to provide timing data, similar to system clock 
116 of host computer 18; the timing data might be required, 
for example, to compute forces output by actuators 64 (e.g., 
forces dependent on calculated velocities or other time 
dependent factors). In alternate embodiments using the USB 
communication interface, liming data for microprocessor 
130 can be retrieved from the USB interface. 

[0078] Local memory 134, such as RAM and/or ROM, is 
preferably coupled to microprocessor 130 in interface 100 to 
store instructions for microprocessor 130 and store tempo- 
rary and other data. Microprocessor 130 may also store 
calibration parameters in a local memory 134 such as an 
EEPROM. Memory 134 may also be used to store the state 
of the force feedback device, including a reference position, 
current control mode or configuration, etc. 

[0079] Sensor interface 136 may optionally be included in 
electronic interface 100 convert sensor signals to signals that 
can be interpreted by the micToprocessor 130 and/or host 
computer system 18. For example, sensor interface 136 can 
receive signals from a digital sensor such as an encoder and 
convert the signals into a digital binary number representing 
the position of a member or component of mechanical 
apparatus 14. An analog to digital converter (ADC) in sensor 
interface 136 can convert a received analog signal to a 
digital signal for microprocessor 130 and/or host computer 
18, Alternately, microprocessor 130 can perform these inter- 
face functions without the need for a separate sensor inter- 
face 136. Or, sensor signals from the sensors can be provided 
directly to host computer system 18. Other types of interface 
circuitry 136 can also be used. For example, an electronic 
interface is described in U.S. Pat. No. 5,576,727, which is 
hereby incorporated by reference herein. 

[0080] Actuator interface 138 can be optionally connected 
between the actuators 64 and microprocessor 130.\Interfacej 
<^i38 converts signals from rnicroprocessor 130 into signals^ 
appropriate to drive the actuate^. Interface 138 can mcludc^ 
^power amplifiers^' switches, digital to analog controllers"^ 
(pAGs), and oilier components. Such interfaces are wjll 
lajownto thgsesMJed in the art. In alternate embodinfents, 
-intcrfaoe 138 cifcmtry caji be provided MtHin Sicrdprocesr 
s6r'136 oiCin the adualors!^ 

[0081] In the described embodiment, power is supplied to 
the actuators 64 and any other components (as required) by 
the USB. Alternatively, power from the USB can be stored 



Jan. 10, 2002 

9 



and regulated by interface 100 or apparatxis 102 and thus 
used when needed to drive actuators 64. Alternatively, a 
power supply 140 can optionally be coupled to actuator 
interface 138 and/or actuators 64 to provide electrical power. 

[0082] Mechanical apparams 102 is coupled to electronic 
interface 100 preferably includes sensors 62, actuators 64, 
and linkage 40. Sensors 62 sense the position, motion, 
and/or other characteristics of mouse 12 along one or more 
degrees of freedom and provide signals to microprocessor 
130 including information representative of those charac- 
teristics. Typically, a sensor 62 is provided for each degree 
of freedom along which mouse 12 can be moved, or, a single 
compound sensor can be used for multiple degrees of 
freedom. Example of sensors suitable for embodiments 
described herein are rotary or linear optical encoders, poten- 
tiometers, non-contact sensors (e.g.. Hall effect magnetic 
sensors, optical sensors, lateral effect photo diodes), velocity 
sensors (e.g., tachometers), or acceleration sensors (e.g., 
accelerometers). Furthermore, either relative or absolute 
sensors can be employed. 

[0083] f/^5ratbts^64^nsmit forces t o mouse - 12"iD^ 
cmore:ddrecdoiis:along^ne^ 

^res^iise^to^ignik^oylp^llbxl'^ic^ 
f^hc^t^^rnpuler"l8r-i;e;^they~are"~"cbm^ 

Typically, an actuator 64 is provided for each degree of 
freedom along which forces are desired to be transmitted. 
Actuators 64 can include two types: active actuators and 
passive actuators. Active actuators include linear current 
control motors, stepper motors, pneumatic/hydraulic active 
actuators, a torquer (motor with limited angular range), a 
voice coil actuator, and other types of actuators that transmit 
a force to an object. Passive actuators can also be used for 
actuators 64, such as magnetic particle brakes, friction 
brakes, pneumatic/hydraulic passive actuators, or passive 
damper elements and generate a damping resistance or 
friction in a degree of motion. In some embodiments, all or 
some of sensors 62 and actuators 64 can be included together 
as a sensor/actuator pair transducer 

[0084] Mechanism 40 is preferably the five-member link- 
age 40 described above, but can also be one of several types 
of mechanisms. For example, mechanisms disclosed in 
co-pending patent applications Ser. Nos. 08/374,288, 
08/400,233, 08/489,068, 08/560,091, 08/623,660, 08/664, 
086, 08/709,012, and 08/736,161, all incorporated by refer- 
ence herein, can be included. Mouse 12 can altematively be 
a puck, joystick, or other device or article coupled to linkage 
40, as described above. 

[0085] Other input devices 141 can optionally be included 
in system 10 and send input signals to microprocessor 130 
and/or host computer 18. Such input devices can include 
buttons, such as buttons 15 on mouse 12, used to supplement 
the input from the user to a GUI, game, simulation, etc. Also, 
dials, switches, sensors, voice recognition hardware (with 
software implemented by host 18), or other input mecha- 
^nisms can be used. 

[0086] Safety or."deadman" switch 150 is preferably 
included in interface device to provide a mechanism to allow 
a user to override and deactivate actuators 64, or require a 
user to activate actuators 64, for safety reasons. Safely 
switch 150 is coupled to actuators 64 such that the user must 
continually activate or close safety switch 150 during 
manipulation of mouse 12 to activate the actuators 64. If, at 



12/08/2003. EAST version: 1.4.1 



us 2002/0003528 Al 



10 



Jan. 10, 2002 



any time, the safely switch is deactivated (opened), power is 
cut to actuators 64 (or the actuators arc oiher>\Tse deacti- 
vated) as long as the safety switch is opened. Safety switch 
150 can be a mechanical or optical switch located on mouse 
12 or on a convenient surface of a housing 26, an electro- 
static contact switch to sense contact of the user, or a 
hand -weight safety switch as described in co-pending patent 
application Ser. No. 08/623,660, incorporated by reference 
herein. The safety switch can be integrated with an indexing 
feature as welt, as described in co-pending patent application 
Ser. No. 08/756,745, incorporated by reference herein. The 
state of the safety switch can be sent to the microprocessor 
130 and/or to host 18. 

[0087] In some embodiments of interface system 10, mul- 
tiple mechanical apparatuses 102 and/or electronic inter- 
faces 100 can be coupled to a single host computer system 
18 through bus 120 (or multiple buses 120) so that multiple 
users can simultaneously interface with the host application 
program (in a multi-player game or simulation, for 
example). In addition, multiple players can interact in the 
host application program with multiple interface systems 10 
using networked host computers 18, as is well known to 
those skilled in the art. Also, the interface device 104 can be 
coupled to multiple host computers; for example, a local 
host computer can display images based on data received 
from a remote host computer coupled to the local host 
through a network. 

[0088] Enhanced Cursor Control and Force Feedback One 
aspect of the present invention is concerned with mouse 
system 10 allowing an enhanced degree of control over a 
cursor for a user, as well as allowing high-fidelity force 
feedback that is not compromised by the control allowed 
over the cursor. The enhanced degree of cursor control 
includes fine positioning of the cursor for target acquiring 
and other tasks, as well as coarse positioning of the cursor 
that is unencumbered and uninhibitied by the fine position- 
ing. 

[0089] Another aspect of the present invention is to allow 
control over the cursor without limits to physical movement 
of the mouse (or other object) to become intrusive to the 
user, i.e., the device incorporates an "indexing" feature that 
corresponds to the case in a non-forcc-fecdback mouse of 
the user repositioning the mouse in its workspace to reduce 
the oSset between the mouse frame and the host computer 
frame. There are several different embodiments described 
herein that include these features. Although the term 
"mouse" is used in the following embodiments, it is intended 
that other types of interface devices and user object may also 
be used with the present invention. In addition, the various 
embodiments presented below are described for use with the 
preferred local microprocessor 130 (or other dedicated pro- 
cessing circuitry on the interface device 104); however, a 
host computer 18 can implement the embodiments of the 
present invention (with any appropriate modifications) if no 
local microprocessor is present in a particular hardware 
embodiment. Alternatively, the host computer can imple- 
ment some functions (such as ballistics calculations and 
indexing calculations) while the microprocessor implements 
other functions. It is assumed in the methods below that host 
computer 18 is displaying a graphical environment such as 
a GUI, game, simulation, etc. on display device 20. 

[0090] The methods described below may be implemented 
with program instructions or code stored on or transferred 



through a computer readable medium. Such a computer 
readable medium may be digital memory chips or other 
memory devices; magnetic media such as hard disk, floppy 
disk, or tape; or other media such as CD-ROM, DVD, 
PCMCIA cards, etc. The computer readable medium may be 
included in the interface device 104, in host computer 18, or 
in both. The program instructions may also be transmitted 
through a channel to interface device 14 from a different 
source. 

[0091] FIG. 5 is a diagrammatic illustration of the local 
frame 30 and display frame 28 and their relationship. The 
local frame 30 is provided in the available workspace in 
which the mouse or other user object may be moved. In the 
embodiment described with reference to FIG. 2, for 
example, the dimensions of the local frame 30 are defined by 
guide opening 76 in the base 42, which may be approxi- 
mately l"xl". Physical limits to the local frame 30 are 
provided by guide pin 78 physically impacting a wall of 
opening 76. The mouse workspace may be defined and 
limited by other mechanisms or structures in other embodi- 
ments. 

[0092] Display frame 28 is shown as a rectangle overlap- 
ping the local frame 30. Display frame 28 is the visible, 
displayed area on display device 20, such as the displayed 
portion of a video screen, on which a user controlled 
graphical object, such as cursor 180, may be moved. In FIG. 
5, the display frame 28 is shown as the same size as local 
frame 30 to emphasize certain concepts in the present 
invention. However, in actuality, the display firame 28 is 
typically larger in actual size than the local frame; for 
example, a computer monitor may have a screen of 15"xll" 
compared to the local frame dimensions VxV\ Thus, move- 
ment in local frame 30 is scaled up to allow movement 
across the entire area of display frame 28. 

[0093] Local frame 30 has a local origin 182 from which 
x and y coordinates of the mouse device in its woricspace are 
referenced. Cursor 180 is shown in FIG. 5 to represent the 
position of both the cursor 180 displayed in display frame 28 
as well as the current position of the mouse 12 in the local 
frame 30 (e.g., the position of axis E and guide pin 78 in the 
embodiment of FIG. 2), where the tip of the cursor indicates 
the precise position. The guide pin 78 (shown as the tip of 
cursor 180) thus has a position of (XJocal, YJocal) in the 
example of FIG. 5. Likewise, display frame 28 has a screen 
origin 184 from which x and y coordinates of the cursor 180 
displayed on the screen 20 are referenced. The cursor 180 
thus has a position of (X screen, Y_screen) in the example 
of FIG. 5. 

[0094] In FIG. 5, the display frame 28 is shown offset 
from local frame 30. This has implications for the indexing 
feature of the present invention, which is described in 
greater detail below. 

[0095] FIG. 6 is a flow diagram illustrating a first method 
200 of the present invention for implementing enhanced 
cursor control and realistic force feedback in mouse system 
10. In method 200, ballistics are provided to allow fine 
positioning and coarse motion of the cursor, but reference 
data is used for determination of position-based forces to 
provide more realistic force feedback. In the preferred 
embodiment, the local microprocessor 130 determines the 
ballistic positions of the cursor, and thus allows the force 
sensations to be calculated based on reference data. 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



11 



Jan. 10, 2002 



[0096] The method begins at 202. In step 204, the mouse 
position in the local frame 30 (i.e., mouse workspace) is read 
by the local microprocessor 130 and is considered the 
"reference position." For example, the position can be 
described as x and y coordinates referenced to workspace 
origin 182. This position is preferably stored in a storage 
area such as local memory 134 and is kept available for 
retrieval by the microprocessor at a later time. 

[0097] In step 206, a ballistic position is determined from 
reference data. "Reference data", as referred to herein, is 
data describing the position and/or motion of the mouse in 
local frame 30. For example, the reference position obtained 
in step 204 is reference data, and the velocity and accelera- 
tion of the mouse in local frame 30 is also reference data. In 
the described embodiment, the ballistic position is deter- 
mined using standard ballistic algorithms and methods based 
on the velocity of the mouse. For example, the current 
velocity of the mouse in the local frame 30 can be deter- 
mined by examining stored positions of the mouse over time 
read by the sensors of the interface device, or by examining 
timing signals or pulses from sensors, or from diflferentiating 
an analog signal, etc. Timing data can be obtained, for 
example, using local clock 132. One preferred embodiment 
uses a haptic accelerator on the interface device 104 to 
determine velocity and/or acceleration of the mouse 12 in its 
local frame 30 and to input such reference data to the local 
microprocessor 130, as disclosed in copending patent appli- 
cation Ser. No. 08/804,535, incorporated by reference 
herein. 

[0098] Once the velocity of the mouse is known, the local 
microprocessor can use a ballistics algorithm to determine 
how to map the position of the cursor to the mouse. A 
"ballistic scale factor" (BSF) can be determined based on the 
mouse velocity, where the BSF is the ratio between move- 
ment of mouse in its workspace and movement of the cursor 
on the screen. The BSF is determined such that low mouse 
velocities create small cursor motions (lower value BSF), 
and large mouse velocities create large, fast cursor motions 
(higher value BSF). The BSF can be determined based on a 
continuous function to determine the precise scaling, or a 
simpler discrete function can be used in which one or more 
velocity thresholds are checked to determine the value of the 
BSF. A change in position of the cursor is then determined 
as the BSF multiplied by the change in reference position (as 
determined using values from step 206). ITie ballistic posi- 
tion is preferably calculated as the old cursor position (the 
position of the cursor in display frame 28 in the last iteration 
of method 200) plus the change in position of the cursor just 
determined. Thus, if the mouse is traveling slowly, then the 
ballistic position is scaled down or not scaled at all since the 
user probably is performing fine positioning of the cursor 
and would like to move the ciusor in small increments. If the 
velocity is large, the ballistic position is scaled higher since 
the user probably is performing coarse movement to get the 
cursor across the screen quickly. Such ballistics scaling is 
well known to those skilled in the art. One or more ballistic 
positions can be referred to as "ballistic data", and the 
ballistic positions are preferably stored by the microproces- 
sor in local memory as the display frame data. ITius, the 
microprocessor keeps track of both local frame data (refer- 
ence data) and display frame data (ballistic data). 

[0099] Alternatively, other methods can be used besides 
ballistics to vary the scaling or the mapping of the cursor 



position to allow fine positioning and coarse motion of the 
cursor. For example, a predictive type of scaling of the 
present invention can be used, which is more "friendly** to 
force feedback implementations than the standard ballistics 
of the prior art. Such predictive scaling only implements a 
fine -positioning scaling that is different from a coarse - 
movement scaling when it is deemed necessary for greater 
control over the cursor. That is, other criteria besides mouse 
velocity are used to determine when to alter the scaling of 
the cursor position .from the mouse position. For example, 
the local microprocessor can examine positions of the mouse 
(or the cursor) over a predetermined period of time to see if 
a fine positioning mode is entered. The microprocessor 
checks whether the cursor has moved completely within a 
small region of predefined size for longer than a predeter- 
mined period of time. The region can be defined by a radius 
or rectangular area surrounding the cursor; for example, a 
region having a radius of (fraction of screen size) can be 
used. The predetermined period of time is some time period 
long enough to indicate that the user is attempting to acquire 
a target or perform some other fine positioning task and may 
be having some difficulty; for example, 3 seconds can be 
used, or the time may depend on the particular task. In 
addition, the cursor should be in motion, since if the cursor 
is still, then the user may simply have taken his or her hand 
off the mouse, and fine positioning mode should not be 
entered. 

[0100] If such conditions apply, then it is assumed/pre- 
dicted that the user needs the assistance of fine positioning 
mode to perform the desired task, and the cursor position is 
set to an adjusted or scaled reference position that has been 
scaled for fine positioning. For example, the cursor position 
can be scaled to the mouse position so that 4 times the 
mouse motion is required to achieve an equivalent cursor 
motion. This allows longer mouse movements to move the 
cursor in shorter increments and greatly assists fine posi- 
tioning of the cursor. Alternatively, a ballistic algorithm can 
be employed in step 364 which makes cursor motion based 
on the velocity of the mouse. Presumably the user is moving 
the mouse slowly so that the cursor motion is scaled down 
according to the ballistics algorithm. If the conditions do not 
apply, the cursor position can be scaled according to a 
constant coarse mapping, since the cursor was not deemed 
to have motion sufficient to change the scahng of the cursor 
for fine positioning. 

[0101] In addition, the microprocessor can check for con- 
ditions to exit the fine positioning mode of the cursor. For 
example, the user may press button 15 on mouse 12 (or other 
input devices) lo manually command the mouse to exit fme 
positioning mode. Or, if the cursor is outside the small 
predefined region (which does not move with the cursor or 
mouse once fine positioning mode is entered) then fine 
positioning mode is exited. Or, if the mouse has remained 
still and unmoving for a minimum predetermined period of 
time in the region, fine positioning mode can be exited. 
Thus, other conditions besides mouse velocity determine 
when different scaling is applied to the cursor position. This 
method is useful for force feedback devices because it 
restricts the variable scaling of cursor position to only cases 
when the cursor is moving within a very small region and is 
close to converging upon a target. Because only small 
motions are involved, the dichotomy between force feed- 
back sensations and the visual motion is not as noticeable to 



12/08/2003, EAST Version: 1.4.1 



us 2002/0003528 Al 



12 



Jan. 10, 2002 



the user. Nevertheless, the force feedback is still preferably 
modified to minimize any distortion, as described below. 

[0102] In step 208, the local microprocessor determines 
interactions between the cursor and the GUI (or other 
graphical environment, such as a 3-D environment) using 
the cursor position. The cursor 180 displayed by the host 
may have encountered or contacted a graphical object or 
region in the GUI that is associated with a force sensation, 
so that the force sensation should be output on the mouse. 
For example, the user may have moved the ciu^r onto an 
icon and held down a button 15 on the mouse to drag the 
icon, which might cause a damping or inertia force to be 
output when the icon is dragged. Or, the cursor may have 
been moved over a window border which causes a spring or 
detent force to be output on the mouse to indicate the 
location of the border. A different situation in which a force 
may be output is when an event has taken place in the GUI 
or other graphical environment which causes a force sensa- 
tion on the mouse. For example, a sound may be output 
indicating a mail message has been received, which in turn 
causes an attractive force on the mouse toward a mail 
program icon. In some cases, multiple force sensations are 
output that are overlaid on each other Many different force 
sensations and associated cursor interactions and events in a 
graphical environment are described in greater detail in 
co-pending patent application Ser. No. 08/571,606, which is 
incorporated by reference herein. In the preferred embodi- 
ment, the local microprocessor 130 is commanded with high 
level host commands from the host computer 18 to imple- 
ment one or more local processes that locally check mouse 
positions and other conditions and output forces when 
particular interactions or events occur in the graphical 
environment. For example, as explained above, the local 
microprocessor can previously be sent a layout of graphical 
objects in the GUI from the host computer to allow the 
microprocessor to check for collisions or interactions. Alter- 
natively, the host computer can check these collisions and 
can send a host command to cause the local processor to 
immediately output a force when the host computer deter- 
mines that such a force is appropriate. 

[0103] It is important to note that the interaction of the 
cursor with other objects in the display frame graphical 
environment is generally determined based on the ballistic 
data from step 206. That is, the cursor position in the 
graphical environment has been scaled according to mouse 
velocity and thus may be different than the position of the 
mouse in the local frame 30, When the local microprocessor 
determines when interactions of the cursor and graphical 
objects occur in step 208 (as is preferred), the local micro- 
processor thus uses the cursor position in the display frame 
28 to determine the location of the cursor and any interac- 
tions of the cursor in the display frame. 

[0104] The process continues to step 210, where indepen- 
dent forces arc calculated. "Independent forces", as referred 
to herein, are those forces which are not based on position 
or motion of the mouse or cursor and thus require neither 
reference data nor ballistic data to be calculated or gener- 
ated. For example, time-based or periodic forces, such as a 
vibration or a jolt, are simply output on the user object at 
predetermined and/or repeating time intervals and durations, 
in specified directions, at a specified starting lime, and at 
predetermined magnitudes. To calculate these forces, in 
contrast with the damping and inertia forces described 



above, neither the position, velocity, or acceleration of the 
mouse is required. Thus, these forces can be calculated 
normally with no need to retrieve reference data or ballistic 
data. 

[0105] In step 212, remaining forces are calculated using 
the reference data. The remaining forces are forces based on 
a position, velocity, and/or acceleration of the mouse 12. For 
example, the calculation of a damping force is generally 
performed using the relation F-Bv, where v is the velocity 
of the object, B is a damping constant, and F is the resulting 
damping force. Here, v is preferably based on the velocity of 
the mouse, so the reference data is used to determine v. 
Similarly, an inertia force uses acceleration in its calculation, 
which is based on the reference data in step 214. Likewise, 
a spring force is typically modelled using F«kx, where x is 
the displacement of the object and k is a spring constant. The 
displacement x is of the mouse and is thus provided using 
the reference data. Also, a friction force can be modelled as 
F=f*(v/|v|), where f is a friction constant, v is the displace- 
ment of the object," provided as reference data, and v/|v| is 
used to indicate the opposite direction to the velocity of the 
mouse (since friction opposes motion). Friction forces can 
be determined in other ways as well. 

[0106] The use of reference data to calculate such posi- 
tion/motion based forces is one of the features of the present 
invention. To effectively reconcile the use of ballistic data 
for cursor positioning with the output of force feedback, the 
present invention uses reference data in the calculation of 
forces while providing ballistic data to the host computer to 
control the position of the cursor on the screen. This allows 
forces to be realistically based on the actual position of the 
mouse in its local frame, yet also allows the cursor to based 
on ballistic data to allow fine positioning and coarse move- 
ment of the cursor in the display frame. If both the cursor 
and the forces were based on reference data, then the 
advantage of more control over cursor motion gained by the 
use of ballistic data would be lost. Thus, the present inven- 
tion advantageously provides ballistic cursor positioning 
while providing forces based on reference data. 

[0107] The local microprocessor 130 allows this use of 
two sets of data to be easily implemented, since the micro- 
processor can keep track of the reference local frame 30 and 
the ballistic display frame 28 separately and can choose data 
from each set as needed. Thus, the microprocessor can select 
ballistic data when determining the location of the cursor in 
relation to other graphical objects in the GUI, can select 
reference data when calculating forces, and can select bal- 
listic data to report to the host computer (described below). 
The use of local microprocessor 130 for local force genera- 
tion allows this implementation. 

[0108] For example, a texture force can make use of both 
frames of data. In one example, a texture can be a pulsating 
force based on the position of the user object, as if dragging 
the user object over bumps or a grating having a particular 
spacing. One way to implement a texture force sensation is 
to output a damping force that is modulated based on mouse 
position, i.e., the damping is turned on and off in sequence 
to simulate bumps at a given spatial frequency. Using the 
two stored frames of data, the damping resistance can be 
generated using the reference data since damping depends 
on the velocity of mouse motion, and the spatial modulation 
of the damping force over the textured region can be based 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



13 



Jan. 10, 2002 



on ballistics data since the position of the cursor on the 
screen dictates when a bump would be felt. 

[0109] Another solution of the present invention is to use 
the ballistic data to both position the cursor and to generate 
all forces. This is a simpler solution in that, once ballistic 
data is determined from the reference data, the reference 
data is no longer needed and the ballistic data can be used 
for all purposes. However, such a solution is not the pre- 
ferred embodiment. This is because, while the ballistic data 
would allow fine positioning and coarse motion of the 
cursor, generation of many types of force sensations with 
ballistic data would distort or diminish the realism of those 
force sensations. For example, damping force sensations are 
typically based on velocity, such as F=Bv. Damping is often 
used to simulate the viscosity of a material, such as the feel 
of moving through a liquid. If the ballistic velocity of the 
cursor is used for v, then the viscosity of a liquid would vary 
depending on whether the mouse is moved slow or fast as 
governed by the ballistics algorithm. The ballistic velocity is 
a scaled velocity, different from the actual velocity of the 
mouse, that would cause a much different magnitude of 
damping than the user expects to feel by moving the mouse. 
A similar situation occiu^ for an inertia force that is based on 
the acceleration of an object. If the ballistic data is used to 
calculate acceleration, a different inertia will be felt than the 
inertia that the user expects by moving the mouse. The 
inertia force would feel as if the moving object were heavier 
when it was moved faster, and lighter when moved slower, 
which is typically undesired. Thus, using reference data for 
the determination of such forces as in method 200 provides 
more realistic forces for the user since the reference data 
describes actual position/motion of the mouse in local frame 
30. The force designer may not want springs to change their 
perceived stiffness, dampers to change their perceived vis- 
cosity, and inertias to change their perceived mass as the 
mapping shifts using a ballistic algorithm, such that using 
the ballistic data in force determination is undesired. 

[0110] In step 214 the forces determined in steps 210 and 
212 (as well as any other forces determined for other 
reasons) are summed and output in step 214. This step may 
be performed at any time after step 212, or concurrently with 
remaining steps 216-222. The total force is output by the 
interface device on the mouse grasped by the user. Actuators 
64 are preferably controlled by the local microprocessor 130 
to output this force, llie output force may also have a 
specified duration, direction, frequency, and other param- 
eters to which the local microprocessor conforms the output 
force. The local microprocessor knows these parameters by 
retrieving them as standard or stored parameters, or may 
receive new parameters directly from the host computer 18. 

[0111] In next step 216, the process checks whether a 
visual spring force was calculated in step 212 (or determined 
to be required in step 208). A visual spring force is special 
in that, when reference data is used to determine forces and 
ballistic data is used to position the cursor on the screen, an 
undesired dichotomy between the displayed spring and the 
feel of the spring results. A "visual spring" force is to be 
distinguished from a "clipped spring" force. The visual 
spring allows the user to see the ciu*sor moving on the screen 
following the expansion or contraction of the spring, and 
feel the spring force as the spring is expanded or contracted. 
The clipped spring allows the user to feel the spring force as 
the mouse is moved, but does not cause the cursor to move 



on the screen. Clipping is described in greater detail below. 
In step 216, other types of forces can also be checked which, 
similar to the visual spring force, cause a dichotomy 
between the visual and haptic experience of the user. 

[0112] If the force sensation is not a visual spring force, 
then in step 218 the cursor position is set equal to the 
ballistic position determined in step 206 above. This allows 
the fine positioning and coarse movement of the cursor as 
described above. Step 222 is then implemented, in which the 
cursor position determined from the above steps is reported 
to the host computer 18, as detailed below. 

[0113] If the force sensation is a visual spring force in step 
216, then the process continues to step 220, where the cursor 
position is based on a constant scale factor. Preferably, the 
scale factor used is the ballistic scale factor that was last in 
effect when the spring force was first applied. That is, while 
the visual spring force is in effect, the scaling of the 
reference data is held constant to the scaling that was 
performed just before the spring force was first output. 

[0114] FIGS, la-lf demonstrate the need to provide a 
constant scaling during the output of a visual spring force. 
These Figures illustrate the situation of a visual spring being 
determined using reference data while the cursor is dis- 
played using ballistic data. FIG. la shows display screen 20 
(display frame 28) of host computer 18 displaying a graphi- 
cal object 240 for simulating a spring force. Object 240 
includes a fixed portion 242 and a moveable portion 244, 
where a simulated spring between portions 242 and 244 is 
shown fully contracted, llie user has moved cursor 180 onto 
moveable portion 244 and has held down the buUon 15 on 
the mouse. FIG. lb shows the equivalent positions to object 
240 in the local frame 30. Starting point 246 is at a 
corresponding position in the mouse workspace to the fixed 
portion 242 of the graphical object 240 on screen 20. The 
position of the mouse 12 itself is indicated by a circle or 
point, which can represent, for example, axis £ or the guide 
pin 78 of the embodiment of FIG, 2. Mouse 12 can be 
positioned to the right of starling point 246 when the spring 
is fully contracted, as shown; alternatively, the mouse 12 can 
be positioned on the starting point 246. 

[0115] FIG. 7c shows display screen 20 displaying object 
240 after the spring has been stretched by the user. As 
indicated by arrow 248, the user has moved cursor 180 
quickly to the right. Since ballistic data is used to display the 
cursor on the screen, the cursor has moved a large distance 
to the right due to the fast motion of the mouse. FIG. Id 
shows an equivalent position of the mouse 12 in the local 
frame. The user feels a spring force pulling the mouse 12 
toward the starting point, represented by spring 249, where 
the spring force magnitude is based on the distance moved 
in the local frame 30. The mouse, of course, has not been 
moved the same actual distance in the local frame 30 as the 
cxirsor has moved on the screen 20 since the cursor motion 
has been scaled higher than the mouse motion. 

[0116] FIG. le shows display screen 20 displaying object 
240 after the user has moved the mouse 12 back toward the 
starting position 246 in the opposite direction to the motion 
of FIGS. 7c and 1± In FIG. le, the mouse is moved slowly 
by the user, as shown by arrow 250. Thus, the ciu-sor 180 
does not move as far due to the ballistic algorithm scaling 
down the cursor movement to allow fine positioning of the 
cursor. However, as shown in FIG. 7/, the mouse has been 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



14 



Jan. 10, 2002 



moved all the way back to the starting point 246. Thus, no 
spring force is output on the mouse 12 because the displace- 
ment of the mouse from the starting point is zero. However, 
when the user looks at the screen 20, the user expects to feel 
a spring force due to the cursor 180 still being some distance 
from the fixed portion 242 of the object 240. This dichotomy 
can be very disconcerling for the user. 

[0117] The present invention solves this dichotomy, as 
explained above, by fixing the mapping between the mouse 
and the cursor during the output of a visual spring force 
sensation, i.e., using a constant scale factor while the spring 
is in effect. Thus, as the cursor 180 is moved quickly to the 
right away from fixed portion 242, the scale factor that was 
used just prior to selecting the object 240 is used and the 
cursor moves an appropriate distance on the screen. Like- 
wise, as the cursor is moved slowly back toward the fixed 
portion 242, the same scale factor is used, resulting in the 
cursor moving the same distance on the screen. In effect, the 
ballistics data is ignored during the output of the spring 
sensation. The force sensation and the visual motion of the 
cursor are thus coordinated and the dichotomy is eliminated. 

[0118] In an alternate embodiment, the cursor can be 
displayed directly according to the reference data (or accord- 
ing to a standard scale factor) rather than according to the 
last scale factor determined from the ballistics algorithm 
before the spring sensation was initiated. 

[0119] Referring back to FIG. 6, after step 220, the 
process continues to step 222, in which the cursor position 
determined from the above steps is sent to the host computer 
18. The "cursor position" is the position which the local 
microprocessor has determined will dictate the position of 
the cursor 180 as it is displayed by the host computer on the 
display device 20 in the display frame 28, The host computer 
receives the cuirsor position and controls the display of the 
cursor at the appropriate location on the display device 20. 
Thus, the host computer simply displays the cursor as if 
receiving input data directly from a peripheral device, and 
preferably remains ignorant of any processing performed on 
the reference data and cursor position by the local processor 
from ballistic, indexing, or other processes. This greatly 
reduces the processing burden on the host computer, since 
the microprocessor performs the ballistics calculations and/ 
or the modifying of the scale factor in step 220 and allows 
the host to simply display the cursor at whatever position is 
reported to it. Hie process 200 then returns to step 204 to 
read another mouse position. 

[0120] Although not described in FIG. 6, the method 200 
may also include any of the indexing features described in 
the present invention. In fact, such an indexing feature is 
preferred since method 200 makes use of ballistics, which 
tend to cause oSisets in the local and display frames as 
described below. Indexing is described in greater detail with 
reference to FIG. 9. The indexing described there can be 
adapted for method 200, by for example, replacing step 206 
with steps 406-416 and adding steps like 424 and 428 to 
determine indexing forces if applicable. The embodiment of 
FIG. 11 can also be adapted for method 200. 

[0121] In addition to modifying the reference data using 
ballistics, indexing, or the scale factor used in step 220 
before sending the cursor position to the host, the local 
microprocessor can also modify the cursor position accord- 
ing to other force feedback features which may be imple- 



mented in the force feedback mouse system 10. For 
example, "clipping" can be used in some situations to 
ptuposely report a cursor position that docs not correspond 
to die mouse position in the local frame 30. Clipping is 
typically used to provide an illusion to the user that a bard 
sm-face is being encountered with the cursor. For example, 
when the cursor 180 is moved against a wall surface, an 
obstruction force having a large enough magnitude force to 
physically stop the user's motion usually cannot be output 
due to actuator limitations, llius, to create the illusion of a 
hard surface, the user is allowed to move the mouse into the 
wall against the obstruction force, but the cursor remains 
displayed against the surface of the wall as if it is impen- 
etrable. Since the user's experience depends heavily on the 
visual motion of the cursor, an illusion of an impenetrable 
wall is maintained. To perform this illusion, the reference 
data from the mouse 12 is "clipped", i.e., modified in that the 
cursor position against the wall is reported to the host by the 
local microprocessor rather than the actual position of the 
mouse through the wall. Dipping may also be performed for 
isometric forces in which a user moves the mouse to control 
a rate control function of a GUI and the cursor remains in a 
constant position. Clipping can be performed in or before 
step 222 if appropriate to modify the cursor position 
reported to the host computer. Clipping is described in 
greater detail in copending patent applications Ser. Nos. 
08/664,086 and 08/756,745, incorporated by reference 
herein. 

[0122] Another force feedback feature that can be used to 
modify the cursor position reported to the host computer is 
"disturbance filtering." Filtering allows the local micropro- 
cessor to filter oscillations and other disturbances out of 
position data before reporting it to the host computer. This 
reduces or eliminates force-feedback-induced dismrbances 
in cursor position that occur as a result of certain force 
sensations, such as vibrations. Thus, the local microproces- 
sor can modify the cursor position to filter out such distur- 
bances and report the filtered cursor position to the host 
computer. Disturbance filtering is described in greater detail 
in co-pending patent application Ser. No. 08/839,249, incor- 
porated by reference herein. 

[0123] In addition, many other steps may also be involved 
in the determination and output of forces and the reporting 
of data to the host which are not necessary to the present 
invention and are thus not detailed herein. 

[0124] FIG. 8 is a flow diagram illustrating a second 
embodiment 300 of the present invention for implementing 
enhanced cursor control and realistic force feedback in 
mouse device 10. In method 300, ballistics are provided to 
allow fine positioning and coarse motion of the cursor, and 
reference data or ballistic data is used for force determina- 
tion to provide more realistic force feedback, 

[0125] The method begins at 302. In step 304, the mouse 
position in the local frame 30 is read by the local micro- 
processor 130 and is the reference position. This position is 
preferably stored in a storage area such as local memory 134 
and is kept available for retrieval by the microprocessor at 
a later time. In step 306, a ballistic position is determined 
from reference data. This is performed similarly to step 206 
descriTjed with reference to FIG. 6. 

[0126] In step 308, the local microprocessor determines 
interactions between the cursor and the GUI (or other 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



15 



Jan. 10, 2002 



graphical environmenl) using the ballistic position. This step 
is similar to step 208 described above with reference to FIG. 
6. The microprocessor also may determine or check for 
events thai cause force sensations. In next step 310, inde- 
pendent forces are calculated, which, as described above, are 
those forces which are not based on position or motion of the 
mouse or cursor and thus require neither reference data nor 
ballistic data to be calculated or generated. 

[0127] In step 312, mouse based forces are calculated 
using the reference data. "Mouse based forces" are those 
force sensations designated to be based on reference data in 
method 300 (and 400), i.e., based on position or motion of 
the mouse rather than on the cursor. In the preferred embodi- 
ment, such force sensations include damping, inertia, and 
friction (based on velocity or acceleration). As explained 
with reference to FIG. 6, these force sensations are more 
realistically modelled using the reference data rather than 
the ballistic data. In addition, "clipped" spring forces are 
preferably mouse-based force sensations (based on mouse 
position). Clipped spring forces differ from visual spring 
forces in that clipped spring forces have no visual compo- 
nent, i.e., the cursor does not move during the output of the 
^spring force. Thus, no possibility of a visual-haptic 
dichotomy exists and tbc spring force can be calculated 
using the reference data. If there arc multiple mouse-based 
force sensations to be output, then those forces are all 
determined using the reference data in step 312 and are 
summed to get a final mouse -based force. 

[0128] In step 314, cursor based forces are calculated 
using ballistic data from slep 306, Cursor based force 
sensations, as referenced herein, are force sensations that are 
preferably determined based on the ballistic data from step 
306 rather than reference data from step 304. Force sensa- 
tions that would cause a undesirable dichotomy between 
what the user sees visually on the display device and what 
the user feels if reference data were used are more likely to 
be characterized as cursor-based force sensations. For 
example, in the preferred embodiment, visual spring forces 
are designated as cursor-based force sensations, since they 
cause a dichotomy between the visual and haptic experi- 
ences of the user as explained above with reference to FIGS. 
la-lf. When a visual spring force is calculated based on 
ballistic data, then both the cursor position and the spring 
force are based on the data in the same display frame 28 and 
no dichotomy occurs. If there are multiple cursor-based 
force sensations to be output, then all such forces are 
determined and summed to get a final cursor-based force. 

[0129] In step 316, the ballistic position determined in step 
306 is reported to the host computer 18 as the cursor 
position. As described with reference to FIG. 6, the cursor 
position is the position which the local microprocessor has 
determined will dictate the position of the cursor 180 as it is 
displayed on the display device 20 in the display frame 28. 
In some situations the reported cursor position may be a 
ballistic position further modified by clipping, disturbance 
filtering, or other processes as explained previously. Step 
316 can be performed at any time once the cursor position 
has been determined, or in parallel with any of steps 308-314 
and 318. 

[0130] In slep 318, the forces determined in steps 310, 
312, and 314 are summed to result in a total force, which is 
then output on mouse 12 by the actuators of the interface 



device. The total force may include other force magnitudes 
contributed by other sources not shown in the above steps. 
The process then returns to step 304 to read another mouse 
position. Thus (as in all the embodiments described herein), 
all three types of forces (independent, mouse-based, and 
cuirsor-based) can be summed together and output on the 
mouse 12 simultaneously. This can occur based on complex 
interactions and events of cursor and graphical environment 
and/or complex types of force sensations. For example, a 
user may stretch a line in a drawing program using the cursor 
180. The stretching function can be implemented by out- 
putting a spring force based on the distance of the stretch, 
and simultaneously outputting a damping force to slow 
down the mouse movement and allow better control. Thus, 
both a mouse-based force (damping) and a cursor-based 
force (spring) would be sununed and output in step 318. 
Independent forces such as jolts might also be summed with 
the other forces and output. 

[0131] The present embodiment (and other embodiments 
herein) thus use both the reference data from the local frame 
30 as well as the ballistic data for the display frame 28 in the 
determination of particular forces to prevent the visual - 
feeling dichotomy explained above. The local microproces- 
sor 130 is well-suited to keep track of data from both frames 
and use data from the appropriate frame as needed. For 
example, an enclosure command is sent from the host to the 
microprocessor 130 which defines an enclosure around a 
window in a GUI. The enclosure has force walls defined 
around the perimeter of the window that obstruct cursor 
movement out of the enclosure. When the cursor is moved 
into the enclosure and is moved against the side of the 
window, the microprocessor uses ballistic data (display 
frame) to detect when the cursor interacts with the window 
side. When the cursor is moved along a side wall of the 
window, a friction force is output by the microprocessor 
based on reference data from the local frame, since friction 
is a mouse-based force. When the user moves the cursor to 
the corner of the window and stretches the window to a new 
size, a spring force is output by the microprocessor based on 
ballistic data from the display frame, since the spring is a 
cursor-based force. Thus, the microprocessor's abihty to 
select data from different frames allows an efficient imple- 
mentation of the present invention. 

[0132] The present embodiment differs from the method 
200 of FIG. 6 in that, in method 300, the determination of 
forces is modified to prevent the visual-feeling dichotomy 
rather than modifying the cursor position reported to the host 
to prevent the dichotomy as in method 200. Visual spring 
forces are characterized as cursor-based force sensations and 
are determined based on ballistic data rather than a fixed 
mapping data or reference data. Method 300 is more efficient 
for the local processor since ballistic data is always reported 
to the host computer (unless modified by other post-pro- 
cesses). This is unlike method 200, where reference data or 
constant-mapping data (not ballistic data) is reported to the 
host computer when outputting a visual spring force and 
ballistic data is reported to the host during the output of other 
types of forces, which requires the local microprocessor to 
monitor the forces and output data from a particular frame 
depending on the force sensation being output. In addition, 
the present method of determining a visual spring force 
based on ballistic data does not distort the spring force 
appreciably and any reduction in force reaUsm is generally 
not noticed by the user of the mouse device. 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



16 



Jan. 10, 2002 



[0133] In addition, many other steps may also be involved 
in the determination and output of forces and the reporting 
of data to the host which are not pertinent to the present 
invention and are not detailed herein. 

[0134] An alternative method to that of FIGS. 6 and 8 for 
solving the visual-feel dichotomy involves outputting only 
independent force sensations by mouse system 10 as 
described with reference to steps 210 and 310. Ballistics can 
be used freely in such an embodiment with no concern over 
distorting the force feedback, since forces are determined 
based only on time and/or other data and are not based on 
position, velocity, and acceleration of the mouse or cursor. 
However, in general, more realistic and immersive forces 
can be implemented using forces based ultimately on the 
position data of the mouse and/or cursor. 

[0135] Also, in some embodiments, both methods of 
FIGS. 6 and 8 (or 9) are available in a single mouse 
interface system, and the user may select which implemen- 
tation he or she wishes based on which one feels better to the 
user. The user may also be able to adjust the strength of the 
ballistics effect by, for example, sending parameters to the 
host computer or interface device. 

[0136] FIG. 9 is a flow diagram illustrating a third, 
preferred embodiment 400 of the present invention for 
implementing enhanced cursor control and realistic force 
feedback in mouse device 10. Method 400 is similar to 
method 300 and includes a preferred embodiment of the 
indexing feature of the present invention. This feature allows 
the user to control the cursor despite any limits to the mouse 
workspace. 

[0137] ITie indexing feature of the present invention 
allows a user to move the cursor throughout the display 
frame of the displayed graphical environment without caus- 
ing the user to experience disconcerting interruptions due to 
the mouse colliding with physical limits to the mouse 
workspace. For example, in the described embodiment of 
FIG. 2, the mouse 12 may be moved in a workspace defined 
by the walls of guide opening 76. When the guide pin 78 
impacts a wall of the opening 76, a limit is reached and the 
user feels the collision with the hard stop. In other force 
feedback mouse implementations, hard stops may also be 
present, since a mechanical linkage that transmits forces 
must have some physical limits to its degrees of freedom. 
The stop prevents the user from moving the mouse further 
in a particular direction and may prevent the user from 
moving the cursor to a desired target in a GUI (an "under- 
reach" situation). These hard stops can also be very discon- 
certing for a user since it interrupts the mouse motion 
abruptly. 

[0138] In a traditional mouse, a user may perform index- 
ing to re-center the mouse in its workspace and reach a 
desired target with the cursor by simply lifting up the mouse 
and placing it closer to the center of a mouse pad or other 
area, and then resuming mouse movement. When the mouse 
is lifted, it stops inputting position data to the host, which 
allows the oflket between mouse and cursor to be reduced. 
However, the force feedback mouse of the described 
embodiment cannot be indexed like a traditional non-force- 
feedback mouse by lifting up and physically recentering the 
mouse in the workspace, since it is attached to a mechanical 
linkage. One solution to this problem is to provide an 
indexing mode, as described in co-pending patent applica- 



tions Scr. No. 08/756,745 filed Nov. 26, 1996 and patent 

application Ser. No, 08/ , filed Jun. 24, 1997 and 

entitled, "Force Feedback Mouse Interface", where the user 
activates a switch or other input device to enter an indexing 
mode that Uiras oflE" the reporting of mouse position (i.e. the 
cursor remains fixed in display frame 28) and the outputting 
offerees while the user re-cenlers the mouse. However, such 
an indexing mode does not address the problem of physical 
impacts: during normal use the force feedback mouse may 
collide with the hard stops frequently, which is far more 
disconcerting for a user than reaching a "soft" limit to mouse 
movement on a mousepad as in a traditional mouse (where 
no actual impact between objects occurs). In addition, the 
hard stops are even more disconcerting and unexpected for 
a user of a force feedback mouse than a traditional mouse, 
since the user expects to experience high-fidelity forces 
based on screen interactions, not a collision with an invisible 
stop. The fidelity of the force environment is corrupted by 
the workspace limits. 

[0139] One way to avoid reaching physical limits to the 
mouse workspace is to report only (scaled) reference data to 
the host computer, thus allowing the mouse to control the 
cursor to all limits of the screen without reaching a limit of 
the mouse workspace. However, as explained herein, such a 
solution does not allow the use of ballistics, which provide 
a greater degree of cursor control. Unfortunately, the use of 
ballistics causes the mouse position in its local frame 30 to 
become oflset from the cursor position in its display frame 
28 and eventually causes the mouse to hit the workspace 
limits. This is simply caused by the variable scaling of cursor 
position based on mouse velocity used in ballistics. For 
example, if a mouse centered in its workspace is moved 
quickly to the right by 0.5 inches from the center point, the 
cursor may be moved 8 inches on the screen away from a 
screen center point. The mouse is then moved back the same 
0.5 inches very slowly and is positioned back al the work- 
space center point. However, the cursor is moved only 1 inch 
back toward the screen center point due to the ballistics 
algorithm, creating an offset between the mouse and cursor 
positions in their respective frames. During more movement, 
these offsets add up, and the mouse may reach a physical 
limit to its workspace before the cursor has reached a desired 
target on the screen. An example of such an ofiket is shown 
in FIG. 5 as the distance between the center of the local 
frame and the center Cg of the screen (display frame). In 
such an example, the mouse can hit the physical border 401 
before the cursor can reach the region 403 on the screen. 
Ofikcts in the local and display frames may also occur even 
when not using ballistics; for example, an application pro- 
gram or operating system may move the cursor indepen- 
dently of the mouse, creating an ofiket and requiring index- 
ing to reduce or eliminate the offset. 

[0140] Thus, other methods must be used to provide a 
force feedback mouse that allows the user greater cursor 
control using ballistics (and to correct frame offsets that 
occur for other reasons) while also allowing an indexing-like 
feature to reduce offsets and preventing the user from 
experiencing collisions with hard physical limits to the 
mouse workspace. Since ballistics are implemented in the 
preferred embodiments, the oflsets between local and dis- 
play frames can become large when controlling a cursor with 
the mouse device over time, and a solution is needed. 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



17 



Jan. 10, 2002 



Method 400 of the present invention implements such a 
solution by providing isometric limits to the mouse work- 
space. 

[0141] Method 400 begins al 402. In step 404, the mouse 
position in the local frame 30 is read by the local micro- 
processor 130 and is the reference position. In step 406, the 
microprocessor checks whether the mouse is within a pre- 
determined distance of a physical limit to the mouse work- 
space. In the described embodiment, this predetermined 
distance is designated an "isometric limit" to the mouse 
workspace. An isometric limit of the present invention 
borders an "isometric region" in the mouse workspace 
which allows a user to control the cursor through isometric 
control rather than isotonic control. As explained with 
reference to FIG. 1, isometric control allows the user to 
control an object based on a rate control paradigm, where an 
amount of the user's pressure in a direction dictates the 
speed of a controlled graphical object in that direction. 
Isotonic control is the normal position control paradigm for 
mouse-cursor mapping, where the position of the mouse in 
its workspace correlates to the position of the controlled 
object in its workspace. 

[0142] Isometric limits 440 and isometric regions 442 of 
the present invention are illustrated in FIG. 5. Local frame 
30 includes a physical limit or border 444 which represents 
the physical limits to movement of the mouse 12 in its 
workspace. For example, in the embodiment of FIG. 2, 
border 444 can be the physical walls to guide opening 76. 
Isometric limits 440 are designated according to software (or 
the equivalent) by the local microprocessor 130 to be at 
some distance d from the border 444; d can be constant 
around the border 444, or d can vary at different sides or 
portions around the workspace. Isometric limits 440 define 
an isometric region 442 which causes an isometric force to 
be output on the mouse 12, as described below. The iso- 
metric region thus borders an isotonic region 443 which 
allows normal isotonic mouse positioning. Preferably, the 
isometric region 442 is an edge region that is fairly small 
compared to the size of the screen; for example, width w of 
the isometric region 442 can be 5% of screen length or width 
or a similar dimension. 

[0143] Referring back to FIG. 9, if the mouse is not past 
an isometric limit 440 (i.e., not within the predetermined 
distance of isometric region 442), then the normal ballistic 
position of the cursor is determined. This includes step 408, 
in which the ballistic screen factor (BSF) is calculated based 
on the mouse velocity, and step 410, in which the change in 
cuxsor position is equal to the BSF limes the change in 
mouse position. Then, in step 412, the cursor position 
(which in this case is the ballistic position) is set equal to the 
old cursor position plus the change in cursor position 
determined in step 410. These steps are described with 
reference to step 206 of FIG. 6. Step 418 is then initiated, 
as described below. 

[0144] If the mouse is past an isometric limit 440 in step 
414, then the indexing feamre of the present invention is 
performed. In step 414, an isometric rate is calculated based 
on the penetration of the mouse into the isometric region. 
The isometric rate determines how fast the cursor is moved 
isometrically based on the amount of compression into the 
virtual spring force (explained below). The greater the 
distance of penetration, the faster the cursor moves. Thus, 



the isometric rate is proportional to the distance of penetra- 
tion. In step 416, the change in cursor position is set equal 
to the isometric rate, and in step 412, the cursor position is 
set equal to the old cursor position plus the change in cursor 
position determined in step 416. Thus, the next position of 
the cursor is determined based on the previous position of 
the cursor and the position of the mouse in the isometric 
region 442. This is the rate control aspect of the isometric 
limits. Since the mouse has moved close to the physical 
border 444 of the workspace, isotonic control of the cursor 
is no longer practical. Instead, isometric (rate) control is 
implemented based on previous cursor positions and the 
direction of mouse motion and the amount of penetration of 
the mouse into region 442. The movement of the cursor 
according to the indexing rate is described in greater detail 
with respect to FIG. lOfe 

[0145] In step 418, the microprocessor determines inter- 
actions between the cursor and the graphical environment 
(such as a GUI) using the determined cursor position. In step 
420, the microprocessor calculates independent forces, and 
in step 422, the microprocessor calculates cursor-based 
forces based on ballistic data. These steps are substantially 
similar to steps 308, 310, and 314, respectively, of method 
300. 

[0146] In step 424, the microprocessor checks if the 
mouse is currently in indexing mode, i.e., whether the mouse 
is in the isometric region 442. If not, then the mouse is in the 
isotonic region 443 and mouse-based forces are calculated 
using reference data. This step is substantially similar to step 
312 of method 300, described above. The process then 
continues to step 432, described below. If the mouse is in 
indexing mode, then in step 428, the microprocessor calcu- 
lates indexing forces. 

[0147] In the described embodiment, the indexing force is 
a resistive spring or restoring force that opposes the mouse's 
motion from the isotonic region 443 to the isometric region 
442. The magnitude and direction of the isometric spring 
force is determined based on the mouse position within the 
limit region 442. For example, the spring force may have a 
magnitude based on the equation F=kx, where k is a spring 
constant and x is the displacement from an origin or starting 
position of the spring. In this implementation, k is a prede- 
termined value and x is the displacement of the mouse from 
the limit 440 into the isometric region 442. The direction of 
the spring force is the direction opposing the motion of the 
mouse toward the physical border 444 of the mouse work- 
space. The isometric spring force simulates a hard surface 
against which the user may exert pressure to control the 
cursor isometrically. 

[0148] For example, FIG. 10a shows a representation of 
local frame 30 in which the position of the mouse in the local 
frame is represented by circle 446. The mouse 446 is moved 
past the limit 450 into the region 442 to a position corre- 
sponding to the circle 448. The distance dl penetrated into 
the region 442 is used in the spring force equation as x to 
determine the magnitude of the opposing spring force. Thus, 
the further the mouse is moved toward the border 444, the 
greater the spring force opposing motion in that direction. If 
the mouse is moved diagonally toward border 444, a diago- 
nal opposing spring force is output. 

[0149] FIG. 10b illustrates the screen 20 of display frame 
28 in which cursor 180 is displayed as controlled by the 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



18 



Jan. 10, 2002 



mouse positions of FIG. 14a, The dashed cursor 452 is the 
position of the cursor when the mouse 12 is at position 446. 
Cursor 452 is displayed far from the left border 454 of the 
screen due to offsets that have added up bc^vee^ mouse and 
cursor position through the use of ballistics. The change in 
cursor position of cursor 452 (as in step 416) is shown as 
cursor 456, which is in the same direction as the motion of 
mouse 12 in region 442 and is a distance d2 from the 
previous position of the cursor. The distance d2 between the 
current and previous positions of the cursor is determined by 
the isometric rate of step 414, determined by distance dl 
penetrated into the region 442 by the mouse; the greater the 
distance dl, the greater the distance d2. Thus, the user can 
control the speed of the movement of cursor 452 toward 
border 454 by pushing against the spring force of region 442 
by a desired pressure magnitude that causes the mouse to 
penetrate the region 442 by a desired distance (the distance 
dl is preferably used as an indication of the magnitude of 
input force from the user — the greater the displacement dl, 
the greater the force or pressure that the user is applying to 
combat the output force). As long as the user pushes the 
mouse 12 against the spring force, the cursor 180 will 
continue to be moved in the appropriate direction at a speed 
controlled by the penetration distance. In this way, the user 
uses isometric control to move the cursor on screen 20 once 
the physical borders 444 are reached by the mouse 12. Once 
the cursor reaches the edge of the display frame 28 in that 
direction, the cursor no longer is moved, even if the mouse 
is still within the isometric region 442. 

[0150] In effect, the microprocessor is changing or con- 
trolling the offset between display frame and local frame 
when determining cursor position in isometric mode. This is 
because the mouse may be fixed in position (or only being 
moved slightly by compressing the spring) in the local frame 
30 while the cursor is moving in the display frame 28. As the 
cursor moves, the offset between frames is reduced. Prefer- 
ably, the local microprocessor stores an "index value" in 
local memory 134 as the positional offset between local and 
display frames and thus may track the offset between frames 
and may performing cursor positioning in an indexing 
situation by changing the index value along a direction 
corresponding to the direction of the penetration of the 
mouse into the isometric region and al a rate of change 
dependent on the depth of penetration into the isometric 
region. This is equivalent to the user in a traditional mouse 
system lifting up the mouse to manually perform indexing 
while the cursor remains fixed on the screen. It should be 
noted that the isometric form of control can be viewed as 
moving the entire local frame 30 while the cursor stays still 
with respect to the local frame. 

[0151] If the user overshoots a desired target using the 
isometric control, then the user can immediately move the 
cursor in the opposite toward the target using isotonic 
position control, since the mouse 12 will have plenty of 
workspace in that direction. For example, if cursor 456 is 
accidentally moved past a target icon 458 so that the cursor 
moves to the left of the icon 458, then the user can move 
mouse 12 to the right to get the cursor back onto the icon. 
Since movement of mouse 12 to the right lakes the mouse 
out of region 442, normal isotonic control of the cursor is 
resumed, 

[0152] In addition, when using isometric regions 442, the 
user does not encounter a hard physical impact of the mouse 



446 colliding with the physical limit 444. The opposing 
spring force generated in connection with region 442 effec- 
tively softens any movement toward limit 444; and, by the 
time the mouse 446 gets very close to limit 444 the spring 
force is usually of high enough magnitude (i.e., dl is large) 
to repel the mouse 446 away from the limit 444 so that an 
impact never occurs or is dramatically softened. This 
removes any disconcerting hard collisions when reaching 
limits to the mouse workspace. 

[0153] Referring to FIG. 9, in step 430, the microproces- 
sor calculates equivalent forces to substitute for the mouse - 
based forces that would have been output in step 426 but for 
the indexing mode. This step aUows forces resulting from 
interactions of the cursor with the graphical environment to 
be felt by the user in indexing mode. Thus, as the user feels 
the spring force from the isometric limit, the user can also 
feel forces overlaid on the spring force caused by interac- 
tions of the cursor with graphical objects, such as a texture 
force when the cursor moves over a "bumpy" region in the 
GUI, or an obstmction force when the cursor impacts a wall. 
However, when the cursor is in rate (isometric) control or 
indexing mode, the mouse may not be moving within its 
workspace; ihe cursor moves at a rate based on the mouse's 
penetration into the isometric region, not based on the 
mouse's movement. Thus, mouse-based forces such as 
damping, inertia, and friction no longer make sense, since 
the mouse is not moving in a position control paradigm. 
Thus, an "equivalent" force to the desired mouse-based 
force is calculated in step 430 to be substituted for the 
mouse-based force, lliis equivalent force may be based on 
timing data or, alternatively, on cursor positions such as 
ballistic data (or other scaled data). For example, timing data 
may be used to calculate an equivalent texture force based 
on a frequency or duration instead of being based on the 
position of the mouse with respect to a bump or divot in the 
GUI. The microprocessor can be sent timing parameters 
from the host that configures the time-based force sensations 
(the duration of a jolt force, the frequency of a vibration 
force, the lime to start outputling the force, etc.) Alterna- 
tively, forces equivalent to mouse-based forces can be based 
on cursor position. A simulated divot or detent force when 
the cursor moves over a border of a window can be calcu- 
lated based on ballistic (cursor) data instead of the usual 
mouse-based reference data when the mouse is in indexing 
mode. 

[0154] In next step 432, the local microprocessor reports 
the cursor position determined in step 412 to the host 
computer, and is similar to step 222 of FIG. 6. In step 434, 
the forces determined in steps 420, 422, 426, 428 and 430, 
if any, are all summed together and output by the actuators 
64 on the mouse 12. Thus, the resistive spring force is output 
if the mouse is in indexing mode as well as any forces 
resulting from interactions in the graphical environment. 
The process then returns to step 404 to read another mouse 
position. 

[0155] In an alternate embodiment, a "hysteresis zone" 
may be provided between the isometric region 442 and the 
isotonic region 443 of the local frame. The hysteresis zone 
creates one location of the limit for entering region 442 and 
a different location of the limit for exiting region 442. For 
example, in FIG. la, when entering region 442 with the 
mouse 12, enter limit 462 can be positioned as limit 440; and 
when the mouse exits region 442, exit limit 464 can be 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



19 



Jan. 10, 2002 



positioned further away from physical border 444. This 
causes the opposing spring force to remain active a further 
distance from the border 444 when exiling and "pxishes" the 
mouse and cursor further from the border. This prevents the 
mouse 12 from being positioned too close to the border and 
allows some isotonic control of the cursor in all directions 
for maximum control when exiting isometric mode, i.e., the 
cursor can be controlled in isotonic mode in a direction 
toward border 444 for the distance d3 upon exiling region 

442. llie hysteresis xone also increases stability of the 
control paradigm, and can enhance the transition from 
isometric control mode to isotonic control mode since the 
spring force will have a larger space/range to gradually 
decrease in magnitude when the mouse is exiting region 442. 

[0156] FIG. 10c illustrates an alternate embodiment of the 
method 400 of FIG, 9 in which ihe isometric limits as 
described above are intended for use as a general positioning 
mechanism for the cursor rather than as an indexing feature 
only for use near the edge of the mouse workspace. This 
allows the offsets between frames to be reduced/corrected 
and also prevents the mouse from hitting physical limits to 
its workspace due to rate control of the cursor and the 
resistive spring force. In addition, this method would not use 
ballistics or other variable scaling of the cursor position and 
thus allows realistic forces to be output. 

[0157] Referring to FIG. 10c, the local frame 30 includes 
physical workspace Hmits 444 which the mouse cannot 
move beyond, as explained above. The central isotonic 
region 443 is defined by dashed line 445, and the surround- 
ing isometric region 442 is defined between the isotonic 
region 443 and the hmit 444. This is similar to the isometric 
limits of method 400 and FIGS. 5 and 10a, except that the 
isotonic region 443 of F[G. 10c is much smaller than the 
equivalent region of FIGS. 5 and li)a. For example, isotonic 
region 443 can have an area that is V* of the workspace area. 

[0158] If the mouse is positioned in the isotonic region 

443, then the cursor position is set equal to (or proportional 
to) the reference position (the cursor position may be scaled 
according to a constant scaling factor). Thus, when the 
mouse is in the isotonic region, cursor positions arc directly 
correlated to the mouse position; no balUstic or other vari- 
able scaling processes modify the position of the cursor. If 
the mouse is positioned in the isometric region, then iso- 
metric mode is entered, which is similar to indexing mode 
described above. The cursor is positioned/moved according 
to an isometric rate, and an isometric spring force opposes 
the mouse's motion into the isometric region. Hie mouse 
position preferably determines the speed of the cursor as it 
moves in the isometric region. The microprocessor also 
overlays any independent and cursor-based forces with the 
indexing force as in method 400. Thus, when the mouse is 
moved into the isometric region against the spring force, the 
cursor is moved according to a rate control paradigm in a 
direction corresponding to the direction of the mouse, where 
the amount of compression of the spring determines the 
speed of the cursor. When the user moves the mouse in the 
opposite direction back into the isotonic region, isotonic 
control is immediately restored and the cursor position once 
more corresponds directly to mouse position. In alternate 
embodiments, a hysteresis effect can be implemented to 
provide entry and exit borders to the isometric region at 
different distances from the physical limit, similar to this 
effect described above. 



[0159] It should be noted that, in alternate embodiments, 
ihe isometric rate control mode described above can be 
activated in other ways for indexing purposes, i.e. to position 
the cursor after a physical limit has been reached by the 
mouse. For example, instead of entering isometric mode 
when the cursor gets close to a limit, the user can simply 
activate isometric mode at any time by pressing and holding 
down a button on the mouse. Some embodiments of such an 
isometric mode are described in co-pending patent applica- 
tion Ser. No. 08/756,745. Such an embodiment would also 
preferably include spring forces on the physical limits 444 of 
the mouse workspace to soften any bard collisions between 
the mouse and the limits. As explained above, in such an 
isometric mode, only non-mouse-based force feedback sen- 
sations (e.g. based on time or cursor position) can be output, 
since mouse-based forces make no sense in a rate control 
paradigm. 

[0160] FIG. 11 is a flow diagram illustrating another 
embodiment 500 of the present invention for implementing 
enhanced cursor control and realistic force feedback in 
mouse device 10. Method 500 includes an alternative 
embodiment of the indexing feature of the present invention 
using an edge scaling feature of the present invention to 
assiu'e that an "under-reach" situation, where the mouse hits 
a physical limit in a direction while the cursor still needs to 
be moved in the corresponding direction on the screen, never 
occurs. 

[0161] Method 500 begins at 502. In step 504, the mouse 
position in the local frame 30 is read by the local micro- 
processor 130 and is the reference position. In step 506, the 
microprocessor checks whether the mouse is within a pre- 
determined distance of the physical workspace limit 444 
(thus deflning a predetermined region next to the physical 
limit) and whether the mouse is moving toward the physical 
hmit 444. The predetermined distance can be a small dis- 
tance or region such as 5% of the total screen dimension in 
that direction. If the mouse is not within this distance or is 
not moving toward limit 444, then indexing is not necessary 
and the process continues to steps 508 and 510, where the 
normal ballistic position of the cursor is determined by 
calculating a ballistic screen factor (BSF) based on the 
mouse velocity and where the change in cursor position is 
equal to the BSF times the change in mouse position, as in 
FIG. 9. In step 512, the cursor position (which in this case 
is the ballistic position) is set equal to the old cursor position 
plus the change in cursor position determined in step 510, 
and step 522 is then initiated, as described below. 

[0162] If the conditions of step 506 are met, the process 
continues to step 514, where the local microprocessor deter- 
mines the distance between the current position of the mouse 
and the physical workspace limit 444 that is closest to the 
current mouse position, i.e., the location of the mouse in the 
predetermined region. In step 516, the local microprocessor 
determines the distance between the current position of the 
cursor and the screen hmit in the display frame 28 that 
corresponds to the closest physical limit to the mouse. For 
example, if the mouse is closest to the right workspace hmit, 
the distance between the mouse and the right Umit is 
determined in step 514 and the distance between the cursor 
and the right screen edge is determined in step 516. If the 
cursor and the mouse have become oQset, the distances 
resulting from steps 514 and 516 can be quite different. In 
step 518, an appropriate scaling factor is determined for the 



12/08/2003. EAST version: 1.4.1 



us 2002/0003528 Al 



20 



Jan. 10, 2002 



cursor position utilizing the distances obtained in steps 514 
and 516. This scaling factor is chosen to allow movement in 
the remaining distance in the mouse workspace to control 
movement of the cursor to the graphical limit of the screen 
20. In other words, the local and display frames are recali- 
brated with respect to each other so that the mouse physical 
limit is not reached before the cursor reaches the screen 
edge, i.e., the microprocessor uses the offset between frames 
to determine a compensating scaling factor. 

[0163] For example, by moving the mouse to the right the 
user will encounter the physical limit 444 at a point where 
the cursor still has 3 inches of screen space to traverse in that 
direction. The local microprocessor thus will deteraaine a 
new scaling factor for use in the predetermined region of 
step 506 that will cause the cursor to reach the end of the 
screen as the mouse is moved through the region, 'lliis is 
accomplished, for example, by finding the ratio between the 
distance found in step 516 and the distance found in step 514 
to be the scaling factor that is multiplied by the mouse 
position to map the remaining mouse worifspace to the 
remaining cursor screen space. 

[0164] In step 520, the change in the cursor position is 
determined based on the new scaling factor ("edge scaling 
factor**), and the process then continues to step 512 where 
the cursor position is determined as the old cursor position 
plus the change in cursor position from step 520. The edge 
scaled cursor position thus is set as the cursor position. 

[0165] In step 522, interactions between cursor and GUI 
are determined; in steps 524, 526, and 528 independent, 
mouse based, and cursor based forces are determined; in step 
530 the cursor position determined in step 512 is reported to 
the host computer, and in step 532 the determined forces are 
summed and the lolal force is output on the mouse 12. ITius, 
steps 522-532 are substantially similar to steps 308-318 of 
FIG. 8, described above. It should be noted that, for cursor 
based forces in step 528 where the cursor is positioned in the 
edge-scaled region, those forces arc calculated based on the 
edge-scaled cursor position, not ballistic cursor positions. 
For example, a visual spring force positioned in the edge 
scaled region would be calculated based on edge-scaled 
data, not ballistic data. Alternatively, the method 200 of 
FIG. 6 can alternatively be used with the indexing steps 506 
and 514-520. 

[0166] Method 500 may cause some problems for the user 
with fine positioning of the cursor within the edge-scaled 
region of the screen, since the cursor motion is scaled higher 
in this region. However, the edge scaling is used only in the 
direction towards the edge of the screen. Thus, if the user 
overshoots a target during the edge scaling, the user may 
move the mouse in the opposite direction to acquire the 
target, at which point normal or ballistic scaling is used 
which typically allows easier fine positioning. 
[0167] Another advantage to the edge scaling process is 
that, since the cursor accelerates when the mouse is posi- 
tioned near the limit of the mouse workspace, the user slows 
down motion of the mouse near the edge to compensate. 
This tends to diminish the likelihood that the mouse will hit 
a physical stop with high speed, and thus less of a collision 
force with the physical stops is felt. In other embodiments, 
the user may wish to feel the hard stops as an indication of 
the cursor hitting the edge of the screen. 

[0168] In an alternate embodiment of the method 500 of 
FIG. 11, steps 506, 508 and 510 are omitted; the distance 



between the current mouse position and the workspace limit 
in the direction of the mouse's movement is determined in 
step 514, and the distance between the cursor position and 
the screen limit corresponding to that physical limit is 
determined in step 516. This allows the local microprocessor 
to calculate a new scaling factor in real time for all positions 
of the mouse in its workspace, not just for regions close to 
the edge of the workspace. For example, the microprocessor 
would always be examining the distance between the current 
mouse position and the workspace limit in step 514 and the 
distance between the cursor and the screen limits in step 516 
and scaling the cursor position accordingly. In one example, 
three "cursor speeds" (i.e., cursor scalings) can be pro\ided: 
coarse, fine, and intermediate. Coarse and fine speeds are 
constant mappings of cursor to mouse position allowing 
different degrees of control However, the intermediate 
speed can use this alternative to method 550 to vary the 
scaling factor according to the offset between local and 
display frames. In an alternative embodiment, the micro- 
processor can determine the distance of the mouse and 
cursor to limits on all sides, such that four different scaling 
factors can be stored and the one that corresponds to the 
cursor's direction is used in step 520. 

[0169] A different embodiment of an indexing feature of 
the present invention for avoiding the cumulative offset 
between the local frame and the display frame is "auto 
centering." This method uses the actuators 64 of the force 
feedback mouse to automatically reduce the offset between 
the local frame and the display frames. When auto centering 
is to be performed, the local microprocessor controls the 
actuators to move the mouse to the location in the local 
frame that corresponds to the center of the display frame, 
thus eliminating the offset between frames. Preferably, auto 
centering is performed when the user is not grasping the 
mouse; otherwise, such movement would confuse the user. 
The auto centering can also be performed only when the 
offset between frames increases over a predetermined 
threshold. Alternatively, a special button, switch, or other 
input device can be provided to the user on mouse 12 or 
other position which would cause the mouse to be auto 
centered when the input device is selected by the user. 

[0170] FIG. 12 is a flow diagram illustrating another 
embodiment 600 of the present invention for providing 
greater cursor control without distorting force feedback. In 
this embodiment, an alternative to traditional ballistics 
called "adaptive resistance" is used to provide enhanced 
cursor control. 

[0171] The method begins at 602. In step 604, the mouse 
position in the local frame 30 is read by the local micro- 
processor 130 and is the reference position. 

[0172] In step 606, the process examines the previous 
positions of the mouse to determine the velocity of the 
mouse. This can be performed similarly to the procedure that 
the ballistics steps of FIG. 6 and FIG. 8 to determine 
velocity of the mouse; for example, the velocity can be 
calculated or simply retrieved from a haptic accelerator or 
other dedicated processing electronics. 

[0173] In step 608, the process determines a resistive force 
magnitude that is inversely based on the mouse velocity. In 
the present embodiment, greater cursor control is provided 
by outputting forces on the mouse 12 to prevent large 
displacements of the mouse and thus the cursor. As in 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



21 



Jan. 10, 2002 



ballisiics algorithms, when the user is moving the mouse 
slowly, it is assumed that the user is performing fine posi- 
tioning of the cursor. '^Thus, for slower velocities of the 
mouse 12, a larger magnitude resistive force is output on the 
mouse, and for higher velocities of the mouse 12, a smaller 
magnitude resistive force is output on the mouse. In one 
embodiment, the magnitude of the resistive force can be 
determined using a discrete fimction that has one or more 
distinct velocity thresholds. For example, if the mouse is 
below a predetermined threshold velocity, a first magnitude 
of damping resistance is selected, and if the mouse is above 
that threshold velocity, a 1-5 second, lower magnitude of 
damping resistance is selected. Alternatively, a continuous 
function can be referenced to provide a continuously-vary- 
ing magnitude based on velocity of the mouse. Either a 
linear or non-linear function can be used. 

[0174] The resistive force determined in step 608 can be 
any of a variety of types of resistive forces. For example, a 
damping force modelled as F«Bv, a friction force modelled 
as F«f*(v/lvD, or a different dissipative force can be used, or 
a combination of two or more forces can be provided. The 
damping constant or friction coefficient can be adjusted as 
described above based on the mouse velocity. Such dissi- 
pative forces slow down and resist quick movement of the 
mouse so that the user will be able to position the cursor 
more slowly and accurately within the graphical environ- 
ment without undesired jitters or overshooting desired tar- 
gets with the cursor. Conversely, when the user moves the 
mouse rapidly, small or zero resistive forces are output to 
allow the user to perform coarse positioning of the cursor. 

[0175] In step 610, the local microprocessor outputs the 
resistive force on the mouse 12 using the actuators of the 
mouse device 10. Preferably, the resistive force that allows 
fine positioning of the cursor is output in all degrees of 
freedom of the mouse, i.e., resistance is felt by the user 
regardless of the direction of the mouse's movement. 

[0176] In step 612, the local microprocessor sends the 
reference position obtained in step 604 to the host computer. 
Step 612 may be performed at any point in the method 600 
or simultaneously with the other steps 606-610. Preferably, 
the reference data is scaled appropriately according to a 
constant mapping to allow mouse motions to control the 
cursor to move to all points displayed on the screen. The 
process then returns to step 604 to read another mouse 
position. 

[0177] Since enhanced cursor control is provided using 
forces and the moxise can control the cursor to all areas of the 
screen without hitting a physical limit, and, since reference 
data and not ballistic data ts reported to the host computer, 
the indexing features of the present invention are not 
required in the embodiment of FIG, 10. However, in some 
embodiments, an indexing feature may still be desired in 
case the local and display frames become ofl&et for some 
reason or to prevent the mouse from colliding hard with a 
physical limit. The indexing features of FIG. 9 or 11 can be 
used, for example. If indexing is used, any indexing force 
may be combined with the resistive force and output in step 
610. 

[0178] FIG. 13 is a flow diagram illustrating another 
embodiment 620 of the present invention for providing an 
enhanced degree of cursor control without distorting force 



feedback. In this embodiment, force "detents" are used to 
assist the user in finely positioning the cursor at a desired 
target. 

[0179] The method begins at 622. In step 624, a high level 
command is received from the host computer indicating the 
location of detent(s) in the local frame 30 and parameters 
describing characteristics of the detent(s). A "detent" is a 
force sensation that assists a user in moving the cursor to a 
particular point or area and reduces the ability of the user to 
move the cursor away from that point or area (such as a 
"snap to" effect). This simulates the feel of a physical detent, 
divot, or valley in a physical surface. For example, when the 
cursor is moved within a predetermined distance of a point, 
an attractive force can be output on the mouse 12 which 
helps guide the cursor to the point. Or, a spring force can be 
provided in a region surrounding a point or area and be 
oriented in a direction toward the point or area, so that when 
the cursor is in the surrounding region, the spring force 
influences the cursor toward the point or region. The spring 
force also has the effect of resisting cursor motion away 
from the point or area once the cursor has acquired the point 
or area. 

[0180] Since detents assist a user in acquiring targets, they 
help the user in fine positioning of the cursor. Hi us a target 
such as an icon can be implemented as a detent with a spring 
force provided in a region surrounding the icon or a center 
point of the icon. It is easier for the user to acquire targets 
that include detents, so the need for ballistics to allow fine 
positioning is much reduced. 

[0181] In the preferred embodiment, the host computer 
sends data to the local microprocessor indicating the loca- 
tions of detent in the graphical environment. Locations of 
detents within the entire graphical environment can be sent, 
or just detents within a predetermined region surrounding 
the present location of the cursor. The host also sends 
parameters indicating the magnitude of the detent force, the 
shape of the detent force or shape of the region where the 
detent exists, the direction of the detent force, etc. Force 
detents are described in greater detail in co-pending patent 
application Ser. No. 08/566,282, which is hereby incorpo- 
rated by reference herein. Step 624 can be performed at any 
time during the process and can be repeated to update the 
local microprocessor's knowledge of the detents in the 
graphical environment. 

[0182] In step 626, the mouse position in the local frame 
30 is read by the local microprocessor 130 and is the 
reference position. In step 628, the microprocessor checks 
whether the mouse is currently at a location to be affected by 
a force detent. The local microprocessor preferably checks 
the detent data sent to it by the host to determine if the mouse 
is positioned al the location of a detent. Thus, the micro- 
processor assumes that the cursor is positioned in the display 
frame 28 (the screen) at a corresponding position to the 
mouse in the local frame 30 according to a constant map- 
ping. If the mouse is not currently at a detent location, the 
process returns to step 626 (or checks for other interactions 
of the cursor with the GUI which may cause forces to be 
output). If the mouse is at the location of the detent, step 630 
is initialed. 

[0183] In step 630, the local microprocessor controls the 
output of a detent force on the mouse 12 by actuators 64 
according lo a locally stored force model and according to 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



22 



Jan. 10, 2002 



any parameters sent by the host computer. In step 632, the 
local microprocessors sends the reference position of the 
mouse obtained in step 624 to the host computer as the 
cm-sor position. As described above, the reference position 
may be scaled according to a constant (non-ballistic) map- 
ping. The process then returns to step 626 to read another 
mouse position. 

[0184] As in the method of FIG. 12, indexing may be used 
in the method 620 of FIG. 13, although it is not necessary 
since ballistic positioning is not used. Modifications for 
indexing are similar as described above. 

[0185] Alternatively, the method of FIG. 13 (or a similar 
method, such as that of FIG. 14) can be implemented using 
force "surfaces" instead of detents to assist in fine position- 
ing of the cursor. An obstruction force simulating the fee! of 
encountering a hard surface such as a wall can be provided 
on appropriate graphical objects or in a region around the 
cursor to help guide the cursor in desired direction and/or 
toward a desired target. For example, a force "enclosure" 
can be provided aroimd a region once the cursor is posi- 
tioned within the region. An enclosure is a box-like object 
having sides characterized by wall and/or texture forces. For 
example, size, location, wall stiffness and width, surface 
texture and friction of the wall, chpping, force characteris- 
tics of the interior region of the enclosure, sctoU surfaces, 
and the speed of the user object necessary to engage the 
forces of the enclosure can all be varied, as described in 
greater detail in co-pending patent apphcation Scr. No. 

08/ , entitled, "Graphical Click Surfaces for Force 

Feedback Apphcations", by Rosenberg et al., filed Jun. 18, 
1997, incorporated by reference herein. 

[0186] Each side of the enclosure would thus resist move- 
ment of the cursor out of the enclosure and would allow the 
user to more easily acquire a target inside the enclosure. In 
other situations, a hard surface on an object can guide the 
cursor along an edge toward a target at the end of the edge. 
Similarly, two surfaces forming a channel can help maintain 
the cursor on a slider bar or other Linear region or object. In 
other embodiments, a field of surfaces or an enclosure can 
be provided around the cursor, whatever its location, when 
the mouse moves in a way to indicate fine positioning is 
desired, as described below with reference to FIG. 14. 

[0187] FIG. 14 is a flow diagram illustrating another 
embodiment 650 for providing an enhanced degree of cursor 
control without distorting force feedback. In this embodi- 
ment, force detents are provided in a region around the 
cursor when the user is believed to need to finely position the 
cursor. 

[0188] The process begins at 652. !n step 654, the mouse 
position in the local frame 30 is read by the local micro- 
processor 130 and is the reference position. In step 656, the 
process examines the previous positions of the mouse to 
determine the velocity of the mouse. This is similar to the 
procedure that the ballistics steps in FIGS. 6 and 8 and step 
606 of FIG. 12 perform to determine velocity. 

[0189] In step 658, the process determines whether the 
mouse velocity is less than a threshold velocity, and whether 
the mouse has been under the threshold velocity for greater 
than a predetermined time period. ITie threshold velocity is 
preferably some small velocity below which the user typi- 
cally desires to finely position the cursor in the graphical 



enviroiunent. The predetermined time period is preferably a 
time period found to typically pass when the user is having 
trouble acquiring a target or performing some other fine 
positioning task (and which can depend on the task). For 
example, a time period of 3 seconds for a particular task 
might be used. In an alternative embodiment, only the 
velocity of the mouse is checked in step 658 and the time 
period is ignored. 

[0190] If the mouse velocity is above the threshold veloc- 
ity or is not under the threshold velocity for the minimum 
time, the process returns to step 654 (of course, forces 
caused by other interactions of the cursor in the GUI or other 
events can be output as described above). If the mouse 
velocity is less than the threshold velocity for the minimum 
time, then the process continues to step 660, where a field of 
multiple force detents are provided in a determined spacing 
over a determined area or region. Thus, the detents are not 
provided if the mouse is moving over the threshold velocity, 
since they would only encumber fast, coarse motion of the 
mouse and cursor. However, if the user is moving the mouse 
slowly for the predetermined lime period, the local proces- 
sor assumes that the user needs assistance in fine position- 
ing, and provided the field of force detents. The detents are 
preferably similar to the detents described with reference to 
FIG. 13, and output forces to slow quick motion of the 
mouse and cursor. The force detents can be provided in a 
rectangular grid, a series of circular radii, or in other 
configurations. These configurations can be predetermined, 
selected by the user, or may vary depending on the nearest 
region or object in the GUI. The field of detents can cover 
the entire screen or display frame, or may be provided only 
in a predefined smaller region surrounding the cursor in a 
predetermined shape or a shape that differs according to the 
region or nearest object of the GUI. In addition, large detents 
or small detents can be provided, and the spacing of the 
detents from each other can be varied as desired. For 
example, a grid of detents can be provided that corresponds 
to a grid of snap points displayed on the screen by a drawing 
program. In a word processor, the detents can correspond to 
letter spacing and line spacing of the current document. Each 
detent can also correspond to each pixel displayed on the 
screen. Ideally, the detents are spaced at the minimum 
resolution required for a give positioning task. For example, 
sensors 62 on the mouse 12 can track 1000 points per square 
inch. This high resolution is not required for the host 
computer, since, for example, 300 pixels are displayed per 
square inch (300 dpi). Thus, detents need only be provided 
at the 300 per square inch resolution. For some tasks, detent 
spacing greater than the pixel spacing can be provided. 

[0191] The local microprocessor can provide the detent 
field entirely independently from the host computer. Alter- 
natively, the host computer can send high level commands 
to enable the force detent feature and to characterize the 
detent spacing, force intensity, and other parameters of the 
detents (thus allowing the user to enable and/or characterize 
detents if desired). 

[0192] In next step 662, the local microprocessor checks 
whether the mouse is at a detent location. If not, the process 
returns to step 654. If so, the local microprocessor controls 
the actuators 64 to output a detent force in step 664, which 
is similar to outpulting forces in the above embodiments. In 
next step 666, the local microprocessor sends the reference 
position (or a position scaled according to a constant map- 



12/08/2003, EAST Version: 1.4.1 



us 2002/0003528 Al 



23 



Jan. 10, 2002 



ping) 10 the host computer as the cursor position (step 666 
can be performed at any time after step 654 or in parallel 
with the other steps). 

[0193] Method 650 is easier to implement than the process 
620 of FIG. 13, since the local microprocessor does not need 
to continually be updated with detent locations in the 
graphical environment from the host computer and does not 
need to check cursor position and the detents in the graphical 
environment to determine if a detent is encountered. Instead, 
the microprocessor need only look at mouse location in the 
local frame and may implement the detents independendy of 
the host. As with FIGS. 12 and 13, indexing is optional in 
this embodiment and may be provided as described above. 

[0194] While this invention has been described in terms of 
several prefened embodiments, it is contemplated that alter- 
ations, permutations and equivalents thereof will become 
apparent to those skilled in the art upon a reading of the 
specification and study of the drawings. For example, 
although examples in a GUI are described, the embodiments 
herein are also very well suited for other two-dimensional 
graphical environments and especially three-dimensional 
graphical environments, where a user would like fine posi- 
tioning in manipulating 3-D objects and moving in a 3-D 
space. For example, the isometric limits are quite helpful to 
move a cursor or controlled object in a 3-D environment 
further than physical limits of the interface device allow. 

[0195] In addition, many different types of forces can be 
applied to the user object 12 in accordance with different 
graphical objects or regions appearing on the computer's 
display screen and which may be mouse -based force sen- 
sations or cursor-based force sensations. Also, the various 
features of the embodiments herein can be combined in 
various ways to provide additional embodiments of the 
present invention. In addition, many types of user objects 
and mechanisms can be provided to transmit the forces to the 
user, such as a mouse, trackball, joystick, stylus, or other 
objects. Furthermore, certain terminology has been used for 
the purposes of descriptive clarity, and not to limit the 
present invention. It is therefore intended that the following 
appended claims include all such alterations, permutations, 
and equivalents as fall within the true spirit and scope of the 
present invention. 



What is claimed is: 

1. A method for providing enhanced cursor control using 
a force feedback interface device coupled to host computer 
displaying a graphical environment and a cursor within said 
graphical environment on a display device, the method 
comprising: 

(a) reading a position of a manipulandum in a device 
workspace as a reference position, said manipulandum 
being grasped by a user; 

(b) reporting a cursor position to said host computer 
derived from said reference position, wherein said host 
computer displays said cursor within said graphical 
environment on said display device at a position cor- 
responding to said cursor position; 

(c) determining whether said cursor interacts with said 
graphical environment as to cause a force to be output 
on said manipulandum; and 



(d) outputting said force on said manipulandum if so 
determined in step (c), 

wherein at least one of said reported cursor position and 
said output force allow said user of said force feedback 
interface device to finely position said cursor within 
said graphical environment and coarsely move said 
cursor as desired in said graphical environment without 
causing a distortion in the quality of forces as expected 
to be experienced by said user from said force feedback 
interface device. 

2. A method as recited in claim 1 wherein said cursor 
position reported to said host computer is said reference 
position that has been modified to allow said fine positioning 
and said coarse movement of said cursor. 

3. A method as recited in claim 2 wherein said cursor 
position is a ballistics position, said ballistics position being 
said reference position modified by a ballistics algorithm in 
which a position of said cursor is mapped to a position of 
said manipulanduna based on a scaling derived from a 
velocity of said manipulandum in said device workspace. 

4. A method as recited in claim 3 wherein said cursor 
position is said reference position modified by a constant 
mapping when said force output in step (c) is a visual spring 
force. 

5. A method as recited in claim 3 wherein said force is 
determined based on motion or position of said manipulan- 
dum with respect to said workspace while said cursor 
position is based on previous cursor positions on said 
display screen. 

6. A method as recited in claim 5 wherein said force is 
determined using said reference position if said force is 
based on motion or position of said manipulandum and is not 
a visual spring force, using said ballistics position if said 
force is a visual spring force, and using neither reference nor 
ballistic position if said force is independent of said motion 
or position of said manipulandum. 

7. A method as recited in claim 1 wherein said force 
output on said manipulandum allows said fine positioning 
and said coarse movement of said cursor within said graphi- 
cal environment. 

8. A method as recited in claim 7 wherein said force is a 
resistive force that resists slow movements of said manipu- 
landum more than it resists fast movements of said manipu- 
landum, said resistive force having a magnitude inversely 
based on a velocity of said manipulandum in said device 
workspace to allow enhanced fine positioning while not 
debilitating coarse movement of said cursor. 

9. A method as recited in claim 7 wherein said force is a 
detent force for guiding said manipulandum to a particular 
position and thereby guiding said cursor to a corresponding 
position in said graphical environment. 

10. A method as recited in claim 9 wherein said force is 
provided by at least one of a plurality of detents arranged in 
a predetermined spacing over a predetermined area, said 
detents being provided when said manipulandum is under a 
predetermined velocity. 

11. A method as recited in claim 9 wherein said force is 
provided by at least one of a plurality of obstruction forces 
simulating surfaces arranged to assist said user to controlling 
said cursor in fine positioning tasks. 

12. A method as recited in claim 1 wherein said manipu- 
landum is a mouse. 

13. A method as recited in claim 1 further comprising 
determining if an indexing feamre should be performed, said 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



24 



Jan. 10, 2002 



indexiDg feature allowing said user to control an ofiket 
between said position of said manipulandun:i in said work- 
space and said position of said cursor on said display screen. 

14. A method for providing enhanced cursor control using 
a force feedback interface device coupled to host computer 
displaying a graphical environment and a cursor within said 
graphical environment on a display device, the method 
comprising: 

(a) reading a position of a manipulandum in a device 
workspace as a reference position, said manipulandum 
being grasped by a user; 

(b) reporting a cursor position lo said host computer, 
wherein said cursor position is said reference position 
that has been modified to a position allowing fine 
positioning or coarse movement of said cursor to 
provide enhanced cursor control to said user when 
moving said manipulandum, and wherein said host 
computer displays said cursor within said graphical 
environment on said display device at a position cor- 
responding to said cursor position; 

(c) determining whether a force is output on said manipu- 
landum based on an interaction of said cursor and said 
graphical environment; 

(d) determining a force, wherein if said force is to be 
determined at least in part based on motion of said 
manipulandum, said reference position is used in said 
determination of said force; and 

(e) outputting said determined force on said manipulan- 
dum. 

15. A method as recited in claim 14 wherein said cursor 
position is a ballistic position, said ballistic position being 
said reference position that has been modified according to 
a ballistics algorithm, said ballistics algorithm allowing fine 
positioning of said cursor at relatively slower movement of 
said manipulandum and allowing coarse movement of said 
cursor at relatively faster movement of said manipulandum. 

16. A method as recited in claim 15 wherein said cursor 
position is based on a constant mapping and not a ballistic 
algorithm when said force is a visual spring force. 

17. A method as recited in claim 15 wherein said force that 
is determined at least in part based on motion of said 
manipulandum is designated as one of two types, said two 
types being a manipulandum-based force sensation that is 
based on at least one reference position of said manipulan- 
dum, and a cursor-based force sensation that is based on at 
least one baUistic position of said cursor. 

18. A method as recited in claim 17 wherein a visual 
spring force is said cursor-based force sensation type, and 
wherein a damping force, inertia force, and friction force are 
said manipulandum-based sensation type. 

19. A method as recited in claim 15 wherein a local 
microprocessor, separate from said host computer and pro- 
vided in said force feedback interface device, stores both 
reference positions and ballistic positions of said manipu- 
landum over time for use in determining said manipulan- 
dum-based force sensations and said cursor based force 
sensations. 

20. A method as recited in claim 19 wherein said local 
microprocessor performs said reporting and determining of 
forces. 



21. A method as recited in claim 20 wherein said force that 
is not based on motion of said manipulandum includes a 
vibration, jolt or other force sensation based solely on time 
parameters. 

22. A method as recited in claim 15 wherein said cursor 
position is further modified by indexing, said indexing 
allowing control over an offset between said position of said 
manipulandum in said workspace and said position of said 
ciusor on said display screen. 

23. A method as recited in 14 wherein said cursor position 
that has been modified is a scaled reference position that 
allows fine positioning of said cursor, said scaling of said 
reference position being performed only when fine position- 
ing is determined to be necessary for positioning said cursor. 

24. A method as recited in claim 23 wherein fine posi- 
tioning is determined to be necessary for positioning said 
cursor when said cursor moves within a region of predeter- 
mined size for longer than a predetermined time period. 

25. A force feedback mouse interface device that provides 
enhanced cursor control over a cursor displayed on a display 
screen of a host computer coupled to said interface device, 
the force feedback interface device comprising: 

a mouse object physically contacted by a user and mov- 
able in a planar workspace in at two degrees of freedom 
with respect to a ground; 

a sensor that reads a position of said mouse object as a 
reference position; 

a plurality of actuators coupled to said mouse object that 
provide a force on said mouse object in said planar 
device workspace; and 

a local microprocessor, separate firom said host computer 
and coupled to said sensor and to said actuators, said 
local microprocessor storing said reference position in 
a local memory and 

reporting a cursor position to said host computer, 
wherein said microprocessor modifies said reference 
position by a ballistics algorithm to determine said 
cursor position, said cursor position providing 
enhanced cursor control to said user when moving 
said mouse object, and wherein said host computer 
displays said cursor within said graphical environ- 
ment on said display device at a position correspond- 
ing to said cursor position, and 

outputting a force based on an interaction of said cursor 
with a graphical environment displayed by said host 
computer, said force being based at least in part on 
motion of said mouse object, wherein said reference 
position is used in said determination of said force. 

26. A force feedback mouse interface device as recited in 
claim 25 wherein said local microprocessor modifies said 
cursor position by a constant scale mapping instead of a 
ballistics algorithm when said force is a visual spring force 
and reports said modified cursor position to said host com- 
puter. 

27. A force feedback mouse interface device as recited in 
claim 25 wherein said local microprocessor uses said cursor 
position to determine said force when said force is a visual 
spring force. 

28. A force feedback mouse interface device as recited in 
claim 25 wherein said local microprocessor modifies said 
ciH^r position for indexing when said mouse position and 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



25 



Jan. 10, 2002 



said cursor position have become ofiket and when said 
mouse object is moved within a predetermined distance of a 
limit to said device workspace. 

29. A method for providing enhanced cursor control using 
a force feedback interface device coupled to a host computer 
displaying a graphical environment and a cursor within said 
graphical environment on a display device, the method 
comprising: 

providing a manipulandum to be grasped and moved by a 
user; 

sensing the motion of said manipulandum with respect to 
a fixed local frame and determining a reference position 
and reference velocity of said manipulandum within 
said local frame; 

deriving a scaled position from said reference position, 
wherein a relation between said scaled position and 
said reference position is influenced by said reference 
velocity in order to allow different scalings for fine 
positioning and course positioning of said cursor, 
thereby providing enhanced cursor control to said user 
when moving said manipulandum; 

using said scaled position in said display of said cursor 
within said graphical environment; 

determining whether a force is to be output on said 
manipulandimi using said scaled position, at least in 
part, in said determination; 

computing a force magnitude of a force to be output on 
said manipulandum using said scaled position, at least 
in part, in said computation of said force magnitude; 
and 

outputting said computed forces to said user through said 
manipulandum. 

30. A method as recited in claim 29 wherein said com- 
puting said force magnitude also uses, in part, said reference 
velocity. 

31. A force feedback interface device for providing 
enhanced cursor control, said interface device being coupled 
to a host computer displaying a graphical environment and 
a cursor within said graphical environment on a display 
device, the force feedback interface device comprising: 

a manipulandum to be grasped and moved by a user; 

a sensor that senses the motion of said manipulandum 
with respect to a fixed local frame and determines a 
reference position and reference velocity of said 
manipulandum within said local frame; 

an actuator operative to output a force on said manipu- 
landum; 

a local microprocessor coupled to said sensor and to said 
actuator, said local microprocessor 

deriving a scaled position from said reference position, 
wherein the relation between scaled position and 
reference position is influenced by said reference 
velocity in order to allow different scalings for fine 
motion and coui^ motion of said cursor, thereby 
providing enhanced cursor control lo said user when 
moving said manipulandum. 



reporting said scaled position to said host computer, 
wherein said host computer uses said scaled position 
in said display of said cursor within said graphical 
environment, 

determining whether a force is to be output on said 
manipulandum, said local processor using said 
scaled position, at least in part, in said determination; 

computing a force magnitude of a force to be output on 
said manipulandum, said local microprocessor using 
said reference position or said reference velocity, at 
least in part, in said computation of said force 
magnitude; and 

outputting said computed force to said user through 
said manipulandum using said actuator. 

32. A force feedback interface device as recited in claim 
41 wherein said local microprocessor also uses said scaled 
position in the computation of said force magnitude. 

33. A force feedback interface device as recited in claim 
39 wherein said local microprocessor stores an index value 
which is the offset between the frame of the reference 
position and the frame of the scaled position. 

34. A method for providing an indexing feature in a force 
feedback mouse device, said mouse device coupled to a host 
computer that displays a cursor and graphical objects in a 
graphical environment on a display screen, said mouse 
device including a mouse being moveable in a local frame, 
wherein said cursor is moved in a display frame based on 
said movement of said mouse, the method comprising: 

determining whether said mouse is within a predeter- 
mined distance to a physical limit of said local frame, 
wherein said predetermined distance is defined by a 
border to a region positioned next to said physical limit; 

determining a location of said mouse in said region; 

using said location to provide control of movement of said 
cursor toward a screen limit of said display frame 
corresponding to said physical limit, wherein a cursor 
position is reported to said host computer that allows 
control of said cursor to an edge of said display frame 
such that said mouse need never contact said physical 
limit of said workspace. 

35. A method as recited in claim 34 wherein said location 
of said mouse in said region is determined by sensing the 
distance of said mouse past a border of said region. 

36. A method as recited in claim 35 further comprising 
outputting a force on said mouse when said mouse moves 
into said region, said force resisting said movement into said 
region. 

37. A method as recited in claim 36 wherein said force is 
a resistive spring force wherein a magnitude of said force is 
based on said distance of said mouse past said region border. 

38. A method as recited in claim 36 wherein said control 
of said movement of said cursor is provided by reporting a 
cursor position to said host computer, wherein said position 
of said cursor on said display screen is based on a previous 
position of said cursor when said mouse is in said region and 
is not based on a corresponding position of said mouse in 
said local frame. 

39. A method as recited in claim 38 wherein said cursor 
position is also based on said distance of said mouse past 
said region border, wherein said distance determines a speed 
of movement of said cursor. 



12/08/2003, EAST version: 1.4.1 



us 2002/0003528 Al 



26 



Jan. 10, 2002 



40. A method as redied in claim 34 wherein said step of 
using said localion includes dclermining a first distance 
between a current position of said mouse and said physical 
limit of said workspace in a direction of motion of said 
mouse. 

41. A method as recited in claim 40 further comprising 
determining a second distance between a current position of 
said cursor and an edge to said display screen, wherein said 
first distance and said second distance are used in determin- 
ing said cursor position. 

42. A method as recited in claim 41 wherein said cursor 
position is based on a scaling of said position of said mouse, 
said scaling based on a ratio between said first and second 
distances, such that said cursor is positioned to the edge of 
said screen area when or before said moiise reaches said 
physical limit to said workspace. 

43. A force feedback mouse device that provides an 
indexing function, said mouse device coupled to a host 
computer that displays graphical objects in a graphical 
environment on a display screen, said mouse device includ- 
ing a mouse being moveable in a mouse workspace, wherein 
said cursor is moved in a screen area based on said move- 
ment of said mouse, and wherein said mouse may control 
motion of said cursor without reaching a physical limit to 
said workspace, said mouse device comprising: 

a mouse object physically contacted by a user and mov- 
able in a planar workspace in at two degrees of freedom 
with respect to a ground; 

a sensor that reads a position of said mouse object; 

a plurality of actuators coupled to said mouse object that 
provide a force on said mouse object in said planar 
device workspace; and 

a local microprocessor, separate from said host computer 
and coupled to said sensor and to said actuators, said 
local microprocessor determining whether said mouse 
is within a predetermined region adjacent to a physical 
limit of said mouse workspace and determining a 
location of said mouse in said region, wherein said 
local microprocessor uses said location to provide 
control of movement of said cursor toward a screen 
limit of said display screen corresponding to said 
physical limit such that said mouse need never contact 
said physical limit of said workspace. 

44. A method for providing an isometric indexing featiu-e 
on a force feedback cursor control interface device, said 
interface device coupled to a host computer that displays 
graphical objects in a graphical environment on a display 
screen, said interface device including a manipulandum 
movable in a physical workspace in order to control the 
position of a cursor displayed by said host computer within 
said graphical environment, the method comprising. 

defining a border region around the outer perimeter of 
said physical workspace of said interface device, 
wherein when said manipulandum is not within said 
border region, said cursor is controlled by said manipu- 
landum through a position control paradigm and when 
said manipulandum is within said border region, said 
cursor is controlled through a rate control paradigm in 
at least one direction of motion; 

determining whether said manipulandum is within said 
border region and determining an amount of penetra- 
tion into said border region; 



outputting a force on said manipulandum opposing said 
penetration into said border region, a magnitude of said 
force being based on the depth of said penetration into 
said border region; and 

using said penetration into said border region to control a 
speed of movement of said cursor along a particular 
direction on said display screen. 

45. A method as recited in claim 44 further comprising 
implementing a hysteresis efifect by removing said force on 
said mouse when said mouse is moved away from said 
physical limit, wherein said force is removed at a distance 
greater from said physical limit than said distance defining 
said border to said region used for outputting said force. 

46. A method for providing an isometric indexing feature 
in a force feedback cursor control interface device, said 
interface device coupled to a host computer that displays 
graphical objects in a graphical environment on a display 
screen, said interface device including a manipulandum 
being movable in a physical workspace in order to control 
the position of a cursor displayed by said host computer 
within said graphical envirormient, the method comprising: 

defining a local frame and a display frame, wherein a 
location of said manipulandum in its workspace is 
referenced with respect to said local frame and wherein 
the location of said cursor with respect to said graphical 
environment is referenced with respect to said display 
frame; 

providing an index value that defines a positional offset 
between said local frame and said display frame; 

defining a border region around an outer perimeter of said 
physical workspace of said interface device; 

determining whether said manipulandum is within said 
border region and determining a penetration into said 
border region; 

outputting a force on said manipulandum opposing said 
penetration into said border region, a magnitude of said 
force being based on a depth of said penetration into 
said border region; and 

using said penetration into said border region to modify 
said index value along a direction corresponding to a 
direction of said penetration and at a rate of change 
dependent upon a depth of penetration into said border 
region. 

47. A method as recited in claim 46 wherein a represen- 
tation of said local frame, said display frame, and said index 
value is stored in memory accessible to a processor local to 
said interface device and separate from said host computer. 

48. A method for providing an indexing function in a force 
feedback mouse device, said mouse device coupled to a host 
computer that displays graphical objects in a graphical 
environment on a display screen, said mouse device includ- 
ing a mouse moveable in a mouse workspace, wherein said 
cursor is moved on a display screen based on said movement 
of said mouse, the method comprising: 

determining a mouse distance between a current position 
of said mouse and a physical limit to said mouse 
workspace; 

determining a cursor distance between a current position 
of said cursor and a displayed edge of said display 
screen; 



12/08/2003, EAST Version: 1.4.1 



us 2002/0003528 Al 



27 



Jan. 10, 2002 



determining a scale factor based on said mouse distance 
and said cursor distance and scaling a position of said 
mouse; 

scaling said position of said mouse to determine a cursor 
position and reporting said cursor position to said host 
computer, said host computer displaying said cursor at 
a position on said display screen corresponding to said 
cursor position, said cursor position allowing said 
cursor to be positioned to an edge of said display screen 



when or before said mouse reaches said physical limit 
to said workspace. 
49. A method as recited in claim 48 further comprising 
determining whether said mouse is within a predetermined 
region adjacent to a physical limit of said mouse workspace, 
wherein said scaling of said mouse position is performed 
only when said mouse is within said predetermined region. 

* 4> « * * 



12/08/2003, EAST Version: 1.4.1